2022-11-23T01:10:21.9798826Z Requested labels: linux.8xlarge.nvidia.gpu
2022-11-23T01:10:21.9798900Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master
2022-11-23T01:10:21.9798922Z Waiting for a runner to pick up this job...
2022-11-23T01:10:22.5785852Z Job is about to start running on the runner: i-0274d02e8f37638bd (organization)
2022-11-23T01:10:27.3150015Z Current runner version: '2.299.1'
2022-11-23T01:10:27.3157489Z Runner name: 'i-0274d02e8f37638bd'
2022-11-23T01:10:27.3158278Z Runner group name: 'Default'
2022-11-23T01:10:27.3159007Z Machine name: 'ip-10-0-0-47'
2022-11-23T01:10:27.3161758Z ##[group]GITHUB_TOKEN Permissions
2022-11-23T01:10:27.3162784Z Actions: write
2022-11-23T01:10:27.3163240Z Checks: write
2022-11-23T01:10:27.3163615Z Contents: write
2022-11-23T01:10:27.3164057Z Deployments: write
2022-11-23T01:10:27.3164563Z Discussions: write
2022-11-23T01:10:27.3164977Z Issues: write
2022-11-23T01:10:27.3165406Z Metadata: read
2022-11-23T01:10:27.3166251Z Packages: write
2022-11-23T01:10:27.3166674Z Pages: write
2022-11-23T01:10:27.3167193Z PullRequests: write
2022-11-23T01:10:27.3167708Z RepositoryProjects: write
2022-11-23T01:10:27.3168144Z SecurityEvents: write
2022-11-23T01:10:27.3168595Z Statuses: write
2022-11-23T01:10:27.3169035Z ##[endgroup]
2022-11-23T01:10:27.3173498Z Secret source: Actions
2022-11-23T01:10:27.3174397Z Prepare workflow directory
2022-11-23T01:10:27.6682189Z Prepare all required actions
2022-11-23T01:10:27.6929107Z Getting action download info
2022-11-23T01:10:27.9278409Z Download action repository 'pytorch/test-infra@main' (SHA:c57ff4d9a93667a5571a80a0e92c3e2674aeedfd)
2022-11-23T01:10:28.2313510Z Download action repository 'pytorch/pytorch@master' (SHA:1cfd3858ac54fe3883534309081631a0a892ba3f)
2022-11-23T01:10:31.4307016Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2022-11-23T01:10:31.7334450Z Getting action download info
2022-11-23T01:10:31.9005364Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2022-11-23T01:10:32.0897684Z Getting action download info
2022-11-23T01:10:32.2421787Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767)
2022-11-23T01:10:32.3792432Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2022-11-23T01:10:32.3794982Z ##[group] Inputs
2022-11-23T01:10:32.3795454Z   build-environment: linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T01:10:32.3797148Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 2, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 3, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 4, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "slow", shard: 1, num_shards: 2, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "slow", shard: 2, num_shards: 2, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "nogpu_AVX512", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "nogpu_NO_AVX2", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "jit_legacy", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "distributed", shard: 1, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 2, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 3, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
]}

2022-11-23T01:10:32.3799112Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:10:32.3799664Z   sync-tag: 
2022-11-23T01:10:32.3800809Z   timeout-minutes: 240
2022-11-23T01:10:32.3801176Z ##[endgroup]
2022-11-23T01:10:32.3802115Z Complete job name: linux-bionic-cuda11.7-py3.10-gcc7 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu)
2022-11-23T01:10:32.4945724Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2022-11-23T01:10:32.4946204Z with:
2022-11-23T01:10:32.4946865Z   github-secret: ***
2022-11-23T01:10:32.4947235Z   activate-with-label: false
2022-11-23T01:10:32.4947562Z   label: with-ssh
2022-11-23T01:10:32.4948074Z   remove-existing-keys: true
2022-11-23T01:10:32.4948409Z env:
2022-11-23T01:10:32.4948700Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:32.4949028Z ##[endgroup]
2022-11-23T01:10:32.6015171Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2022-11-23T01:10:32.6245026Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-11-23T01:10:32.6245460Z with:
2022-11-23T01:10:32.6246155Z   submodules: recursive
2022-11-23T01:10:32.6246504Z   fetch-depth: 0
2022-11-23T01:10:32.6246795Z env:
2022-11-23T01:10:32.6247129Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:32.6247513Z ##[endgroup]
2022-11-23T01:10:32.6536000Z ##[group]Run retry () {
2022-11-23T01:10:32.6536405Z retry () {
2022-11-23T01:10:32.6536780Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-11-23T01:10:32.6537144Z }
2022-11-23T01:10:32.6537479Z echo "${GITHUB_WORKSPACE}"
2022-11-23T01:10:32.6537866Z if [ -z "${NO_SUDO}" ]; then
2022-11-23T01:10:32.6538242Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T01:10:32.6538595Z else
2022-11-23T01:10:32.6538943Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T01:10:32.6539270Z fi
2022-11-23T01:10:32.6539642Z mkdir "${GITHUB_WORKSPACE}"
2022-11-23T01:10:32.6558449Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:10:32.6558840Z env:
2022-11-23T01:10:32.6559164Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:32.6559502Z   NO_SUDO: 
2022-11-23T01:10:32.6559798Z ##[endgroup]
2022-11-23T01:10:32.6686352Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:10:35.4841050Z ##[group]Run malfet/checkout@silent-checkout
2022-11-23T01:10:35.4841464Z with:
2022-11-23T01:10:35.4841793Z   ref: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:10:35.4842126Z   fetch-depth: 0
2022-11-23T01:10:35.4842436Z   submodules: recursive
2022-11-23T01:10:35.4842759Z   quiet-checkout: true
2022-11-23T01:10:35.4843097Z   repository: pytorch/pytorch
2022-11-23T01:10:35.4843604Z   token: ***
2022-11-23T01:10:35.4843918Z   ssh-strict: true
2022-11-23T01:10:35.4844232Z   persist-credentials: true
2022-11-23T01:10:35.4844561Z   clean: true
2022-11-23T01:10:35.4844860Z   lfs: false
2022-11-23T01:10:35.4845157Z   set-safe-directory: true
2022-11-23T01:10:35.4845472Z env:
2022-11-23T01:10:35.4846066Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:35.4846386Z ##[endgroup]
2022-11-23T01:10:35.6353516Z Syncing repository: pytorch/pytorch
2022-11-23T01:10:35.6355487Z ##[group]Getting Git version info
2022-11-23T01:10:35.6356087Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T01:10:35.6356795Z [command]/usr/bin/git version
2022-11-23T01:10:35.6357135Z git version 2.37.1
2022-11-23T01:10:35.6364811Z ##[endgroup]
2022-11-23T01:10:35.6384460Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/242fe719-e1aa-4b05-8d81-5773fddff4a9' before making global git config changes
2022-11-23T01:10:35.6385636Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T01:10:35.6391819Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:10:35.6436009Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T01:10:35.6442674Z ##[group]Initializing the repository
2022-11-23T01:10:35.6446478Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:10:35.6479505Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-11-23T01:10:35.6480032Z hint: is subject to change. To configure the initial branch name to use in all
2022-11-23T01:10:35.6480533Z hint: of your new repositories, which will suppress this warning, call:
2022-11-23T01:10:35.6480909Z hint: 
2022-11-23T01:10:35.6481351Z hint: 	git config --global init.defaultBranch <name>
2022-11-23T01:10:35.6481896Z hint: 
2022-11-23T01:10:35.6482337Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-11-23T01:10:35.6482926Z hint: 'development'. The just-created branch can be renamed via this command:
2022-11-23T01:10:35.6483312Z hint: 
2022-11-23T01:10:35.6483752Z hint: 	git branch -m <name>
2022-11-23T01:10:35.6484343Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-11-23T01:10:35.6494578Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-11-23T01:10:35.6528434Z ##[endgroup]
2022-11-23T01:10:35.6529023Z ##[group]Disabling automatic garbage collection
2022-11-23T01:10:35.6532948Z [command]/usr/bin/git config --local gc.auto 0
2022-11-23T01:10:35.6563850Z ##[endgroup]
2022-11-23T01:10:35.6565051Z ##[group]Setting up auth
2022-11-23T01:10:35.6574501Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T01:10:35.6609592Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T01:10:35.6910561Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T01:10:35.6943986Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T01:10:35.7241875Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T01:10:35.7402531Z ##[endgroup]
2022-11-23T01:10:35.7403125Z ##[group]Fetching the repository
2022-11-23T01:10:35.7411868Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-11-23T01:11:28.9394671Z [command]/usr/bin/git rev-parse --verify --quiet 1cfd3858ac54fe3883534309081631a0a892ba3f^{object}
2022-11-23T01:11:28.9424355Z 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:11:28.9430311Z ##[endgroup]
2022-11-23T01:11:28.9430904Z ##[group]Determining the checkout info
2022-11-23T01:11:28.9432021Z ##[endgroup]
2022-11-23T01:11:28.9432578Z ##[group]Checking out the ref
2022-11-23T01:11:28.9436898Z [command]/usr/bin/git checkout --quiet --force 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:11:30.6702419Z ##[endgroup]
2022-11-23T01:11:30.6703067Z ##[group]Setting up auth for fetching submodules
2022-11-23T01:11:30.6708530Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T01:11:30.6764723Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-11-23T01:11:30.6798444Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-11-23T01:11:30.6830400Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-11-23T01:11:30.6862468Z ##[endgroup]
2022-11-23T01:11:30.6863019Z ##[group]Fetching submodules
2022-11-23T01:11:30.6867809Z [command]/usr/bin/git submodule sync --recursive
2022-11-23T01:11:30.7195286Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-11-23T01:11:30.7509185Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-11-23T01:11:30.7511528Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-11-23T01:11:30.7514500Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-11-23T01:11:30.7517653Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-11-23T01:11:30.7520995Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-11-23T01:11:30.7524664Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2022-11-23T01:11:30.7528928Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-11-23T01:11:30.7532605Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-11-23T01:11:30.7536487Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-11-23T01:11:30.7540424Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-11-23T01:11:30.7544744Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-11-23T01:11:30.7548882Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2022-11-23T01:11:30.7553249Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-11-23T01:11:30.7557942Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-11-23T01:11:30.7562480Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-11-23T01:11:30.7567617Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-11-23T01:11:30.7572696Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-11-23T01:11:30.7577748Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:11:30.7582889Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-11-23T01:11:30.7588400Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-11-23T01:11:30.7593926Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-11-23T01:11:30.7599501Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-11-23T01:11:30.7605350Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-11-23T01:11:30.7611741Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-11-23T01:11:30.7617757Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-11-23T01:11:30.7623827Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-11-23T01:11:30.7630126Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-11-23T01:11:30.7636377Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-11-23T01:11:30.7643072Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-11-23T01:11:30.7652636Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-11-23T01:11:30.7659413Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-11-23T01:11:30.7666208Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-11-23T01:11:30.7673328Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-11-23T01:11:30.7680394Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-11-23T01:11:30.7688285Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-11-23T01:11:30.7695615Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-11-23T01:11:30.7703153Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-11-23T01:11:30.7710769Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-11-23T01:11:30.7718610Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-11-23T01:11:30.7726945Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-11-23T01:11:30.7735468Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-11-23T01:11:30.7764629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-11-23T01:11:31.0590918Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-11-23T01:11:31.2661018Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-11-23T01:11:31.4734199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-11-23T01:11:31.7782624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-11-23T01:11:32.0716022Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2022-11-23T01:11:34.3224012Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-11-23T01:11:40.0900247Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-11-23T01:11:40.4995828Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-11-23T01:11:41.0813293Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-11-23T01:11:42.7278315Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-11-23T01:11:44.1978365Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2022-11-23T01:11:45.9935419Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-11-23T01:11:53.5307597Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-11-23T01:11:54.4065268Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-11-23T01:11:56.0069546Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-11-23T01:11:57.2315146Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-11-23T01:11:57.4545165Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-11-23T01:11:57.9815514Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-11-23T01:11:58.3225741Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-11-23T01:11:59.3714551Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-11-23T01:11:59.7974647Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-11-23T01:12:00.0168479Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-11-23T01:12:00.3566449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-11-23T01:12:02.3473483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-11-23T01:12:04.1517910Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-11-23T01:12:04.5885261Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-11-23T01:12:11.5570995Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-11-23T01:12:13.2931235Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-11-23T01:12:13.7619854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-11-23T01:12:14.0551171Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-11-23T01:12:20.4492860Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-11-23T01:12:20.6902395Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-11-23T01:12:20.9372018Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-11-23T01:12:21.8142099Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-11-23T01:12:22.0546230Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-11-23T01:12:22.4093733Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-11-23T01:12:22.7297541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-11-23T01:12:23.3256750Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-11-23T01:12:25.9570498Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-11-23T01:12:26.5212735Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-11-23T01:12:28.9051064Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-11-23T01:12:28.9177109Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-11-23T01:12:28.9274580Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-11-23T01:12:28.9564679Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-11-23T01:12:28.9838405Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-11-23T01:12:29.0280675Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2022-11-23T01:12:29.7856266Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-11-23T01:12:29.8110102Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-11-23T01:12:29.9325124Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2022-11-23T01:12:29.9723487Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-11-23T01:12:30.3326083Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2022-11-23T01:12:30.8492106Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2022-11-23T01:12:31.1482794Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-11-23T01:12:31.2034045Z Submodule path 'third_party/fbgemm': checked out '4d1738b3142a6cb0c032cd639e239566010b054a'
2022-11-23T01:12:31.2052205Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:12:31.2055318Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:12:31.2058615Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:12:31.2062077Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:12:31.2089895Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-11-23T01:12:32.2351980Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-11-23T01:12:32.8203724Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-11-23T01:12:33.8548552Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-11-23T01:12:34.2249029Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-11-23T01:12:34.3489189Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-11-23T01:12:34.4200301Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-11-23T01:12:34.4313881Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-11-23T01:12:34.5465202Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-11-23T01:12:34.5889455Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b'
2022-11-23T01:12:34.5991480Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-11-23T01:12:34.6464859Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-11-23T01:12:34.6754059Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195'
2022-11-23T01:12:34.7306498Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-11-23T01:12:34.7435584Z Submodule path 'third_party/ideep': checked out '5ddc65efe0428bbce2942b3ce5e3ce15239abe2f'
2022-11-23T01:12:34.7452190Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-11-23T01:12:34.7477890Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-11-23T01:12:43.8918416Z Submodule path 'third_party/ideep/mkl-dnn': checked out 'd19d0f795c60695bd32f894c6f01771b2dfbe24d'
2022-11-23T01:12:43.8938791Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:12:43.8966602Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-11-23T01:12:53.2078069Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '650085b2f3643aad05c629425983491d63b5c289'
2022-11-23T01:12:53.2195998Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-11-23T01:12:53.2372090Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-11-23T01:12:53.3497431Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562'
2022-11-23T01:12:53.3515926Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:12:53.3519103Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:12:53.3546199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-11-23T01:12:54.5189415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-11-23T01:12:55.7306574Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-11-23T01:12:55.7961624Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-11-23T01:12:55.8208983Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2022-11-23T01:12:55.8380159Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-11-23T01:12:55.9690132Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-11-23T01:12:56.2962057Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-11-23T01:12:56.2994988Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-11-23T01:12:56.2998308Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-11-23T01:12:56.3026602Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-11-23T01:12:56.7401726Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-11-23T01:12:57.6697935Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-11-23T01:12:57.7076902Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-11-23T01:12:57.7255977Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-11-23T01:12:57.7273120Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:12:57.7299764Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-11-23T01:12:59.5747591Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-11-23T01:12:59.5772639Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:12:59.5775780Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:12:59.5803551Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-11-23T01:13:00.0181457Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-11-23T01:13:01.9541586Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-11-23T01:13:02.0338890Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-11-23T01:13:02.0356427Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:02.0385619Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-11-23T01:13:02.4076971Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-11-23T01:13:02.4182705Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-11-23T01:13:02.7348026Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-11-23T01:13:02.7371462Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:02.7374887Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:02.7402412Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-11-23T01:13:03.2279545Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-11-23T01:13:04.3019554Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-11-23T01:13:04.3850486Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-11-23T01:13:04.3948125Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-11-23T01:13:04.4072716Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-11-23T01:13:04.4472791Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917'
2022-11-23T01:13:04.4573648Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-11-23T01:13:04.4911165Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-11-23T01:13:04.5016548Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-11-23T01:13:04.5552176Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-11-23T01:13:04.6903605Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-11-23T01:13:04.7220632Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-11-23T01:13:04.7237741Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:04.7240961Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:04.7244195Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:04.7247987Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:04.7275182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-11-23T01:13:05.7665475Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-11-23T01:13:06.0638533Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-11-23T01:13:07.5125903Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-11-23T01:13:08.5201386Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-11-23T01:13:08.5397119Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-11-23T01:13:08.6185150Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-11-23T01:13:08.6514157Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-11-23T01:13:08.6531033Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:08.6557350Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-11-23T01:13:08.8875509Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-11-23T01:13:09.0472241Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-11-23T01:13:09.0513981Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-11-23T01:13:09.0842563Z Entering 'android/libs/fbjni'
2022-11-23T01:13:09.0885051Z Entering 'third_party/FP16'
2022-11-23T01:13:09.0929421Z Entering 'third_party/FXdiv'
2022-11-23T01:13:09.0973224Z Entering 'third_party/NNPACK'
2022-11-23T01:13:09.1017347Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:09.1061491Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:09.1105914Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:09.1160768Z Entering 'third_party/benchmark'
2022-11-23T01:13:09.1204278Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:09.1249452Z Entering 'third_party/cub'
2022-11-23T01:13:09.1293138Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:09.1343435Z Entering 'third_party/cutlass'
2022-11-23T01:13:09.1394469Z Entering 'third_party/eigen'
2022-11-23T01:13:09.1441494Z Entering 'third_party/fbgemm'
2022-11-23T01:13:09.1484994Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:09.1528000Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:09.1571211Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:09.1613921Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:09.1659605Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:09.1705637Z Entering 'third_party/fmt'
2022-11-23T01:13:09.1749183Z Entering 'third_party/foxi'
2022-11-23T01:13:09.1792585Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:09.1836289Z Entering 'third_party/gloo'
2022-11-23T01:13:09.1880227Z Entering 'third_party/googletest'
2022-11-23T01:13:09.1923898Z Entering 'third_party/ideep'
2022-11-23T01:13:09.1968948Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:09.2016728Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:09.2069253Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:09.2115114Z Entering 'third_party/ittapi'
2022-11-23T01:13:09.2160048Z Entering 'third_party/kineto'
2022-11-23T01:13:09.2203487Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:09.2248322Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:09.2292403Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:09.2338590Z Entering 'third_party/neon2sse'
2022-11-23T01:13:09.2382485Z Entering 'third_party/nlohmann'
2022-11-23T01:13:09.2428698Z Entering 'third_party/onnx'
2022-11-23T01:13:09.2486382Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:09.2529599Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:09.2574435Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:09.2617320Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:09.2665571Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:09.2708798Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:09.2751803Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:09.2801126Z Entering 'third_party/pocketfft'
2022-11-23T01:13:09.2845530Z Entering 'third_party/protobuf'
2022-11-23T01:13:09.2893356Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:09.2936857Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:09.2981192Z Entering 'third_party/psimd'
2022-11-23T01:13:09.3024947Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:09.3069176Z Entering 'third_party/pybind11'
2022-11-23T01:13:09.3113148Z Entering 'third_party/python-enum'
2022-11-23T01:13:09.3157217Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:09.3200728Z Entering 'third_party/python-six'
2022-11-23T01:13:09.3244085Z Entering 'third_party/sleef'
2022-11-23T01:13:09.3288893Z Entering 'third_party/tbb'
2022-11-23T01:13:09.3334616Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:09.3378270Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:09.3420883Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:09.3462620Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:09.3505455Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:09.3548684Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:09.3594445Z Entering 'third_party/zstd'
2022-11-23T01:13:09.3652232Z ##[endgroup]
2022-11-23T01:13:09.3653139Z ##[group]Persisting credentials for submodules
2022-11-23T01:13:09.3657322Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-11-23T01:13:09.3981702Z Entering 'android/libs/fbjni'
2022-11-23T01:13:09.4024453Z Entering 'third_party/FP16'
2022-11-23T01:13:09.4067236Z Entering 'third_party/FXdiv'
2022-11-23T01:13:09.4111188Z Entering 'third_party/NNPACK'
2022-11-23T01:13:09.4154293Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:09.4197787Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:09.4240727Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:09.4296019Z Entering 'third_party/benchmark'
2022-11-23T01:13:09.4338529Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:09.4380992Z Entering 'third_party/cub'
2022-11-23T01:13:09.4424003Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:09.4474416Z Entering 'third_party/cutlass'
2022-11-23T01:13:09.4527227Z Entering 'third_party/eigen'
2022-11-23T01:13:09.4575185Z Entering 'third_party/fbgemm'
2022-11-23T01:13:09.4618691Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:09.4660389Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:09.4702693Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:09.4756932Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:09.4800029Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:09.4845646Z Entering 'third_party/fmt'
2022-11-23T01:13:09.4888045Z Entering 'third_party/foxi'
2022-11-23T01:13:09.4931093Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:09.4974028Z Entering 'third_party/gloo'
2022-11-23T01:13:09.5015995Z Entering 'third_party/googletest'
2022-11-23T01:13:09.5059292Z Entering 'third_party/ideep'
2022-11-23T01:13:09.5100118Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:09.5143495Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:09.5191837Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:09.5234696Z Entering 'third_party/ittapi'
2022-11-23T01:13:09.5276918Z Entering 'third_party/kineto'
2022-11-23T01:13:09.5318374Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:09.5360240Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:09.5402713Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:09.5444758Z Entering 'third_party/neon2sse'
2022-11-23T01:13:09.5487144Z Entering 'third_party/nlohmann'
2022-11-23T01:13:09.5530408Z Entering 'third_party/onnx'
2022-11-23T01:13:09.5585704Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:09.5628238Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:09.5671865Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:09.5713568Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:09.5759957Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:09.5801438Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:09.5842366Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:09.5888967Z Entering 'third_party/pocketfft'
2022-11-23T01:13:09.5929877Z Entering 'third_party/protobuf'
2022-11-23T01:13:09.5975546Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:09.6018810Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:09.6061482Z Entering 'third_party/psimd'
2022-11-23T01:13:09.6103303Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:09.6144123Z Entering 'third_party/pybind11'
2022-11-23T01:13:09.6186291Z Entering 'third_party/python-enum'
2022-11-23T01:13:09.6227252Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:09.6269161Z Entering 'third_party/python-six'
2022-11-23T01:13:09.6310582Z Entering 'third_party/sleef'
2022-11-23T01:13:09.6352526Z Entering 'third_party/tbb'
2022-11-23T01:13:09.6396643Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:09.6437880Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:09.6478706Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:09.6519388Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:09.6560788Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:09.6601322Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:09.6645870Z Entering 'third_party/zstd'
2022-11-23T01:13:09.6701178Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-11-23T01:13:09.7019087Z Entering 'android/libs/fbjni'
2022-11-23T01:13:09.7058795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-11-23T01:13:09.7076327Z Entering 'third_party/FP16'
2022-11-23T01:13:09.7115291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-11-23T01:13:09.7133351Z Entering 'third_party/FXdiv'
2022-11-23T01:13:09.7173820Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-11-23T01:13:09.7190999Z Entering 'third_party/NNPACK'
2022-11-23T01:13:09.7229195Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-11-23T01:13:09.7246934Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:09.7285083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-11-23T01:13:09.7303232Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:09.7342746Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2022-11-23T01:13:09.7360076Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:09.7398396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-11-23T01:13:09.7428185Z Entering 'third_party/benchmark'
2022-11-23T01:13:09.7467281Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:09.7485332Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:09.7524369Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-11-23T01:13:09.7542481Z Entering 'third_party/cub'
2022-11-23T01:13:09.7580674Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-11-23T01:13:09.7598209Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:09.7638106Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-11-23T01:13:09.7661365Z Entering 'third_party/cutlass'
2022-11-23T01:13:09.7699112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2022-11-23T01:13:09.7722898Z Entering 'third_party/eigen'
2022-11-23T01:13:09.7761817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-11-23T01:13:09.7781497Z Entering 'third_party/fbgemm'
2022-11-23T01:13:09.7822302Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-11-23T01:13:09.7839693Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:09.7878679Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-11-23T01:13:09.7896013Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:09.7934237Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-11-23T01:13:09.7951714Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:09.7991089Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:09.8008332Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:09.8046328Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-11-23T01:13:09.8064836Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:09.8104612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-11-23T01:13:09.8124406Z Entering 'third_party/fmt'
2022-11-23T01:13:09.8163347Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-11-23T01:13:09.8180709Z Entering 'third_party/foxi'
2022-11-23T01:13:09.8219888Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-11-23T01:13:09.8237208Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:09.8275493Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-11-23T01:13:09.8293544Z Entering 'third_party/gloo'
2022-11-23T01:13:09.8333059Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-11-23T01:13:09.8350712Z Entering 'third_party/googletest'
2022-11-23T01:13:09.8390204Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:09.8407881Z Entering 'third_party/ideep'
2022-11-23T01:13:09.8445381Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-11-23T01:13:09.8462584Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:09.8500970Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-11-23T01:13:09.8520361Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:09.8560374Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-11-23T01:13:09.8585328Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:09.8624421Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-11-23T01:13:09.8641467Z Entering 'third_party/ittapi'
2022-11-23T01:13:09.8680302Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-11-23T01:13:09.8698214Z Entering 'third_party/kineto'
2022-11-23T01:13:09.8737173Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-11-23T01:13:09.8754362Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:09.8792848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-11-23T01:13:09.8810701Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:09.8849714Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:09.8868024Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:09.8907298Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-11-23T01:13:09.8924842Z Entering 'third_party/neon2sse'
2022-11-23T01:13:09.8963616Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-11-23T01:13:09.8980563Z Entering 'third_party/nlohmann'
2022-11-23T01:13:09.9018935Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-11-23T01:13:09.9038352Z Entering 'third_party/onnx'
2022-11-23T01:13:09.9077859Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-11-23T01:13:09.9109203Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:09.9148696Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:09.9166468Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:09.9205208Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:09.9225273Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:09.9263998Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-11-23T01:13:09.9280923Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:09.9320597Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-11-23T01:13:09.9343940Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:09.9383600Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:09.9401770Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:09.9441174Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:09.9459285Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:09.9499570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-11-23T01:13:09.9522439Z Entering 'third_party/pocketfft'
2022-11-23T01:13:09.9561918Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-11-23T01:13:09.9579502Z Entering 'third_party/protobuf'
2022-11-23T01:13:09.9618750Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-11-23T01:13:09.9639479Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:09.9678087Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:09.9695694Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:09.9733855Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:09.9753103Z Entering 'third_party/psimd'
2022-11-23T01:13:09.9792117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-11-23T01:13:09.9809866Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:09.9848563Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-11-23T01:13:09.9865601Z Entering 'third_party/pybind11'
2022-11-23T01:13:09.9904663Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:09.9922204Z Entering 'third_party/python-enum'
2022-11-23T01:13:09.9960457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-11-23T01:13:09.9978270Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:10.0016737Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-11-23T01:13:10.0033850Z Entering 'third_party/python-six'
2022-11-23T01:13:10.0072311Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-11-23T01:13:10.0091068Z Entering 'third_party/sleef'
2022-11-23T01:13:10.0130824Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-11-23T01:13:10.0148553Z Entering 'third_party/tbb'
2022-11-23T01:13:10.0187393Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-11-23T01:13:10.0207054Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:10.0246461Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-11-23T01:13:10.0264457Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:10.0303141Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:10.0320180Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:10.0358498Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-11-23T01:13:10.0377033Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:10.0415099Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-11-23T01:13:10.0432904Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:10.0470881Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:10.0487916Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:10.0525934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-11-23T01:13:10.0546099Z Entering 'third_party/zstd'
2022-11-23T01:13:10.0585308Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-11-23T01:13:10.1573038Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-11-23T01:13:10.1889217Z Entering 'android/libs/fbjni'
2022-11-23T01:13:10.1932350Z Entering 'third_party/FP16'
2022-11-23T01:13:10.1975436Z Entering 'third_party/FXdiv'
2022-11-23T01:13:10.2018218Z Entering 'third_party/NNPACK'
2022-11-23T01:13:10.2060678Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:10.2104056Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:10.2147031Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:10.2200909Z Entering 'third_party/benchmark'
2022-11-23T01:13:10.2244068Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:10.2287665Z Entering 'third_party/cub'
2022-11-23T01:13:10.2330244Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:10.2379480Z Entering 'third_party/cutlass'
2022-11-23T01:13:10.2427895Z Entering 'third_party/eigen'
2022-11-23T01:13:10.2473465Z Entering 'third_party/fbgemm'
2022-11-23T01:13:10.2516882Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:10.2558373Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:10.2601279Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:10.2643779Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:10.2686271Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:10.2732570Z Entering 'third_party/fmt'
2022-11-23T01:13:10.2776625Z Entering 'third_party/foxi'
2022-11-23T01:13:10.2819290Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:10.2861690Z Entering 'third_party/gloo'
2022-11-23T01:13:10.2905676Z Entering 'third_party/googletest'
2022-11-23T01:13:10.2949702Z Entering 'third_party/ideep'
2022-11-23T01:13:10.2991114Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:10.3035352Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:10.3084741Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:10.3128073Z Entering 'third_party/ittapi'
2022-11-23T01:13:10.3171910Z Entering 'third_party/kineto'
2022-11-23T01:13:10.3214894Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:10.3258187Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:10.3302475Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:10.3346472Z Entering 'third_party/neon2sse'
2022-11-23T01:13:10.3388439Z Entering 'third_party/nlohmann'
2022-11-23T01:13:10.3432443Z Entering 'third_party/onnx'
2022-11-23T01:13:10.3487903Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:10.3530380Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:10.3574921Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:10.3616310Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:10.3663669Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:10.3707092Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:10.3749512Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:10.3796190Z Entering 'third_party/pocketfft'
2022-11-23T01:13:10.3839284Z Entering 'third_party/protobuf'
2022-11-23T01:13:10.3886184Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:10.3928071Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:10.3972674Z Entering 'third_party/psimd'
2022-11-23T01:13:10.4015221Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:10.4057387Z Entering 'third_party/pybind11'
2022-11-23T01:13:10.4099616Z Entering 'third_party/python-enum'
2022-11-23T01:13:10.4141569Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:10.4184389Z Entering 'third_party/python-six'
2022-11-23T01:13:10.4226801Z Entering 'third_party/sleef'
2022-11-23T01:13:10.4269032Z Entering 'third_party/tbb'
2022-11-23T01:13:10.4313461Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:10.4356280Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:10.4398459Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:10.4439491Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:10.4482638Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:10.4578661Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:10.4623505Z Entering 'third_party/zstd'
2022-11-23T01:13:10.4678715Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-11-23T01:13:10.5000274Z Entering 'android/libs/fbjni'
2022-11-23T01:13:10.5042559Z Entering 'third_party/FP16'
2022-11-23T01:13:10.5085504Z Entering 'third_party/FXdiv'
2022-11-23T01:13:10.5127956Z Entering 'third_party/NNPACK'
2022-11-23T01:13:10.5170776Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:10.5213985Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:10.5257072Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:10.5310300Z Entering 'third_party/benchmark'
2022-11-23T01:13:10.5352760Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:10.5396579Z Entering 'third_party/cub'
2022-11-23T01:13:10.5438951Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:10.5487980Z Entering 'third_party/cutlass'
2022-11-23T01:13:10.5538604Z Entering 'third_party/eigen'
2022-11-23T01:13:10.5583973Z Entering 'third_party/fbgemm'
2022-11-23T01:13:10.5627692Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:10.5670068Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:10.5714094Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:10.5755584Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:10.5798810Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:10.5843625Z Entering 'third_party/fmt'
2022-11-23T01:13:10.5886347Z Entering 'third_party/foxi'
2022-11-23T01:13:10.5928245Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:10.5970152Z Entering 'third_party/gloo'
2022-11-23T01:13:10.6012960Z Entering 'third_party/googletest'
2022-11-23T01:13:10.6056838Z Entering 'third_party/ideep'
2022-11-23T01:13:10.6098887Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:10.6143630Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:10.6192996Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:10.6235481Z Entering 'third_party/ittapi'
2022-11-23T01:13:10.6278201Z Entering 'third_party/kineto'
2022-11-23T01:13:10.6321204Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:10.6363749Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:10.6408255Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:10.6450614Z Entering 'third_party/neon2sse'
2022-11-23T01:13:10.6493313Z Entering 'third_party/nlohmann'
2022-11-23T01:13:10.6536759Z Entering 'third_party/onnx'
2022-11-23T01:13:10.6591839Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:10.6634898Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:10.6678861Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:10.6721759Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:10.6769873Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:10.6812781Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:10.6855193Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:10.6902551Z Entering 'third_party/pocketfft'
2022-11-23T01:13:10.6944851Z Entering 'third_party/protobuf'
2022-11-23T01:13:10.6990698Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:10.7032733Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:10.7075955Z Entering 'third_party/psimd'
2022-11-23T01:13:10.7118898Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:10.7162771Z Entering 'third_party/pybind11'
2022-11-23T01:13:10.7205253Z Entering 'third_party/python-enum'
2022-11-23T01:13:10.7246824Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:10.7289820Z Entering 'third_party/python-six'
2022-11-23T01:13:10.7332380Z Entering 'third_party/sleef'
2022-11-23T01:13:10.7375057Z Entering 'third_party/tbb'
2022-11-23T01:13:10.7421298Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:10.7464718Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:10.7507059Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:10.7548988Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:10.7591365Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:10.7632877Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:10.7678534Z Entering 'third_party/zstd'
2022-11-23T01:13:10.7730901Z ##[endgroup]
2022-11-23T01:13:10.7782220Z [command]/usr/bin/git log -1 --format='%H'
2022-11-23T01:13:10.7813205Z '1cfd3858ac54fe3883534309081631a0a892ba3f'
2022-11-23T01:13:10.7958308Z Prepare all required actions
2022-11-23T01:13:10.7990532Z ##[group]Run ./.github/actions/setup-linux
2022-11-23T01:13:10.7990871Z env:
2022-11-23T01:13:10.7991176Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:10.7991486Z ##[endgroup]
2022-11-23T01:13:10.8060641Z ##[group]Run set -euo pipefail
2022-11-23T01:13:10.8061013Z set -euo pipefail
2022-11-23T01:13:10.8061348Z function get_ec2_metadata() {
2022-11-23T01:13:10.8061735Z   # Pulled from instance metadata endpoint for EC2
2022-11-23T01:13:10.8062260Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-11-23T01:13:10.8062709Z   category=$1
2022-11-23T01:13:10.8063083Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-11-23T01:13:10.8063447Z }
2022-11-23T01:13:10.8063759Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-11-23T01:13:10.8064179Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-11-23T01:13:10.8064606Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-11-23T01:13:10.8064992Z echo "system info $(uname -a)"
2022-11-23T01:13:10.8078531Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:10.8078891Z env:
2022-11-23T01:13:10.8079197Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:10.8079514Z ##[endgroup]
2022-11-23T01:13:10.8184464Z ami-id: ami-096198a0bccc6bad4
2022-11-23T01:13:10.8249558Z instance-id: i-0274d02e8f37638bd
2022-11-23T01:13:10.8311675Z instance-type: g3.8xlarge
2022-11-23T01:13:10.8319886Z system info Linux ip-10-0-0-47.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-11-23T01:13:10.8339665Z ##[group]Run if systemctl is-active --quiet docker; then
2022-11-23T01:13:10.8340111Z if systemctl is-active --quiet docker; then
2022-11-23T01:13:10.8340499Z     echo "Docker daemon is running...";
2022-11-23T01:13:10.8340843Z else
2022-11-23T01:13:10.8341207Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-11-23T01:13:10.8341582Z fi
2022-11-23T01:13:10.8353155Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:10.8353494Z env:
2022-11-23T01:13:10.8353790Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:10.8354115Z ##[endgroup]
2022-11-23T01:13:10.8406552Z Docker daemon is running...
2022-11-23T01:13:10.8426456Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-11-23T01:13:10.8426987Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-11-23T01:13:10.8427444Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:13:10.8428019Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-11-23T01:13:10.8428567Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-11-23T01:13:10.8440885Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:10.8441252Z env:
2022-11-23T01:13:10.8441558Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:10.8441898Z   AWS_RETRY_MODE: standard
2022-11-23T01:13:10.8442213Z   AWS_MAX_ATTEMPTS: 5
2022-11-23T01:13:10.8442544Z   AWS_DEFAULT_REGION: us-east-1
2022-11-23T01:13:10.8442870Z ##[endgroup]
2022-11-23T01:13:11.7781869Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-11-23T01:13:11.7782443Z Configure a credential helper to remove this warning. See
2022-11-23T01:13:11.7783133Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-11-23T01:13:11.7783682Z 
2022-11-23T01:13:11.7783916Z Login Succeeded
2022-11-23T01:13:11.7819424Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:13:11.7819896Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:13:11.7820432Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:13:11.7833631Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:11.7833970Z env:
2022-11-23T01:13:11.7834264Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:11.7834581Z ##[endgroup]
2022-11-23T01:13:11.7929511Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2022-11-23T01:13:11.7929946Z with:
2022-11-23T01:13:11.7930511Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:13:11.7931037Z env:
2022-11-23T01:13:11.7931353Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:11.7931686Z ##[endgroup]
2022-11-23T01:13:11.7949513Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:13:11.7949965Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:13:11.7950399Z # ignore output since only exit code is used for conditional
2022-11-23T01:13:11.7950861Z # only pull docker image if it's not available locally
2022-11-23T01:13:11.7951338Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-11-23T01:13:11.7951811Z   retry docker pull "${DOCKER_IMAGE}"
2022-11-23T01:13:11.7952163Z fi
2022-11-23T01:13:11.7964473Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:11.7964845Z env:
2022-11-23T01:13:11.7965164Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:11.7966287Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:13:11.7966881Z ##[endgroup]
2022-11-23T01:13:12.0301428Z 072aae4a77ed7d3a69ad5683420509c41301b940: Pulling from pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7
2022-11-23T01:13:12.0301964Z a404e5416296: Pulling fs layer
2022-11-23T01:13:12.0302311Z 42d021f37342: Pulling fs layer
2022-11-23T01:13:12.0302646Z 9dab8401a678: Pulling fs layer
2022-11-23T01:13:12.0304568Z 2fc72180b8eb: Pulling fs layer
2022-11-23T01:13:12.0305252Z 16e6a4d496ed: Pulling fs layer
2022-11-23T01:13:12.0305972Z 3a4c0b092a9e: Pulling fs layer
2022-11-23T01:13:12.0306525Z d9e19734e968: Pulling fs layer
2022-11-23T01:13:12.0306850Z 8859cc6b3ab7: Pulling fs layer
2022-11-23T01:13:12.0307176Z 1a4b44db2103: Pulling fs layer
2022-11-23T01:13:12.0307513Z fac00e927cfe: Pulling fs layer
2022-11-23T01:13:12.0307838Z 2133f4081ddb: Pulling fs layer
2022-11-23T01:13:12.0308178Z 3ceac802dd07: Pulling fs layer
2022-11-23T01:13:12.0308520Z 69c929400d35: Pulling fs layer
2022-11-23T01:13:12.0309305Z bc2be817cb7e: Pulling fs layer
2022-11-23T01:13:12.0309930Z d04926f1c68d: Pulling fs layer
2022-11-23T01:13:12.0310372Z 91f116f19c0d: Pulling fs layer
2022-11-23T01:13:12.0310710Z a7cf5b3894f8: Pulling fs layer
2022-11-23T01:13:12.0311050Z d9e19734e968: Waiting
2022-11-23T01:13:12.0311369Z 722cddd823f9: Pulling fs layer
2022-11-23T01:13:12.0311684Z 2e882087d824: Pulling fs layer
2022-11-23T01:13:12.0312015Z 8859cc6b3ab7: Waiting
2022-11-23T01:13:12.0312326Z 2133f4081ddb: Waiting
2022-11-23T01:13:12.0312649Z ba6235196410: Pulling fs layer
2022-11-23T01:13:12.0315579Z fac00e927cfe: Waiting
2022-11-23T01:13:12.0316988Z 313c5ee380ab: Pulling fs layer
2022-11-23T01:13:12.0317699Z 3ceac802dd07: Waiting
2022-11-23T01:13:12.0318286Z 6ff0fc00b0a9: Pulling fs layer
2022-11-23T01:13:12.0318631Z 06f043acdbfb: Pulling fs layer
2022-11-23T01:13:12.0318980Z a8c562f6a1cf: Pulling fs layer
2022-11-23T01:13:12.0319314Z 1a4b44db2103: Waiting
2022-11-23T01:13:12.0319629Z 4468cd4f574c: Pulling fs layer
2022-11-23T01:13:12.0319954Z 69c929400d35: Waiting
2022-11-23T01:13:12.0320279Z 46e5d3cec398: Pulling fs layer
2022-11-23T01:13:12.0332542Z bc2be817cb7e: Waiting
2022-11-23T01:13:12.0333232Z 165006759af3: Pulling fs layer
2022-11-23T01:13:12.0333936Z 2c660101e145: Pulling fs layer
2022-11-23T01:13:12.0334529Z 8ce731125a4e: Pulling fs layer
2022-11-23T01:13:12.0335202Z 102ddcd90753: Pulling fs layer
2022-11-23T01:13:12.0335691Z 0cff5716a932: Pulling fs layer
2022-11-23T01:13:12.0336032Z d04926f1c68d: Waiting
2022-11-23T01:13:12.0336340Z 2fc72180b8eb: Waiting
2022-11-23T01:13:12.0336662Z 91f116f19c0d: Waiting
2022-11-23T01:13:12.0337178Z 867a6ccb577f: Pulling fs layer
2022-11-23T01:13:12.0337521Z 863c35620b44: Pulling fs layer
2022-11-23T01:13:12.0337847Z a8c562f6a1cf: Waiting
2022-11-23T01:13:12.0338176Z f828f00c6d66: Pulling fs layer
2022-11-23T01:13:12.0338488Z 165006759af3: Waiting
2022-11-23T01:13:12.0338801Z 16e6a4d496ed: Waiting
2022-11-23T01:13:12.0339110Z 2c660101e145: Waiting
2022-11-23T01:13:12.0339423Z f0fe61569b0b: Pulling fs layer
2022-11-23T01:13:12.0339761Z 06f043acdbfb: Waiting
2022-11-23T01:13:12.0340076Z 313c5ee380ab: Waiting
2022-11-23T01:13:12.0340381Z 102ddcd90753: Waiting
2022-11-23T01:13:12.0340709Z a9f3d4742233: Pulling fs layer
2022-11-23T01:13:12.0341047Z 000b6751ea6f: Pulling fs layer
2022-11-23T01:13:12.0341363Z a7cf5b3894f8: Waiting
2022-11-23T01:13:12.0341692Z 023a41fa48e6: Pulling fs layer
2022-11-23T01:13:12.0342016Z 722cddd823f9: Waiting
2022-11-23T01:13:12.0342328Z 081025f05026: Pulling fs layer
2022-11-23T01:13:12.0342669Z 5970defc1d8b: Pulling fs layer
2022-11-23T01:13:12.0343002Z 8ce731125a4e: Waiting
2022-11-23T01:13:12.0343367Z 7e2d6313145f: Pulling fs layer
2022-11-23T01:13:12.0343695Z 4b4d66451d67: Pulling fs layer
2022-11-23T01:13:12.0344035Z 75f1ead35ace: Pulling fs layer
2022-11-23T01:13:12.0344366Z 4468cd4f574c: Waiting
2022-11-23T01:13:12.0344674Z 793c37004dab: Pulling fs layer
2022-11-23T01:13:12.0345014Z 4f1313d71da0: Pulling fs layer
2022-11-23T01:13:12.0345357Z 6386b2adbe28: Pulling fs layer
2022-11-23T01:13:12.0345671Z ba6235196410: Waiting
2022-11-23T01:13:12.0346004Z f21c39caec7a: Pulling fs layer
2022-11-23T01:13:12.0346337Z 863c35620b44: Waiting
2022-11-23T01:13:12.0346638Z 3a4c0b092a9e: Waiting
2022-11-23T01:13:12.0346972Z 2baaae32e3d5: Pulling fs layer
2022-11-23T01:13:12.0347319Z 128ebe6811c5: Pulling fs layer
2022-11-23T01:13:12.0347648Z 772fa4efddc3: Pulling fs layer
2022-11-23T01:13:12.0347991Z c36ac12376c5: Pulling fs layer
2022-11-23T01:13:12.0348318Z 023a41fa48e6: Waiting
2022-11-23T01:13:12.0348629Z 85cc9b957510: Pulling fs layer
2022-11-23T01:13:12.0348969Z 9ab0826d88b5: Pulling fs layer
2022-11-23T01:13:12.0349295Z 081025f05026: Waiting
2022-11-23T01:13:12.0349591Z 2e882087d824: Waiting
2022-11-23T01:13:12.0349905Z c36ac12376c5: Waiting
2022-11-23T01:13:12.0350221Z 75f1ead35ace: Waiting
2022-11-23T01:13:12.0350516Z 85cc9b957510: Waiting
2022-11-23T01:13:12.0350831Z 2baaae32e3d5: Waiting
2022-11-23T01:13:12.0351149Z f0fe61569b0b: Waiting
2022-11-23T01:13:12.0351447Z 46e5d3cec398: Waiting
2022-11-23T01:13:12.0351760Z f828f00c6d66: Waiting
2022-11-23T01:13:12.0352074Z 4b4d66451d67: Waiting
2022-11-23T01:13:12.0352390Z 0cff5716a932: Waiting
2022-11-23T01:13:12.0352689Z 772fa4efddc3: Waiting
2022-11-23T01:13:12.0353000Z 7e2d6313145f: Waiting
2022-11-23T01:13:12.0353308Z 793c37004dab: Waiting
2022-11-23T01:13:12.0353600Z 128ebe6811c5: Waiting
2022-11-23T01:13:12.0353915Z f21c39caec7a: Waiting
2022-11-23T01:13:12.1632758Z 42d021f37342: Verifying Checksum
2022-11-23T01:13:12.1633177Z 42d021f37342: Download complete
2022-11-23T01:13:12.2442608Z 2fc72180b8eb: Verifying Checksum
2022-11-23T01:13:12.2443050Z 2fc72180b8eb: Download complete
2022-11-23T01:13:12.3837060Z a404e5416296: Verifying Checksum
2022-11-23T01:13:12.3837708Z a404e5416296: Download complete
2022-11-23T01:13:12.3926395Z 16e6a4d496ed: Download complete
2022-11-23T01:13:12.4707075Z d9e19734e968: Download complete
2022-11-23T01:13:12.5766175Z 9dab8401a678: Verifying Checksum
2022-11-23T01:13:12.5766838Z 9dab8401a678: Download complete
2022-11-23T01:13:12.6658129Z 1a4b44db2103: Verifying Checksum
2022-11-23T01:13:12.6658736Z 1a4b44db2103: Download complete
2022-11-23T01:13:12.7366605Z fac00e927cfe: Verifying Checksum
2022-11-23T01:13:12.7367069Z fac00e927cfe: Download complete
2022-11-23T01:13:13.1842187Z a404e5416296: Pull complete
2022-11-23T01:13:13.4808157Z 42d021f37342: Pull complete
2022-11-23T01:13:14.4026958Z 9dab8401a678: Pull complete
2022-11-23T01:13:14.5175883Z 2fc72180b8eb: Pull complete
2022-11-23T01:13:14.6322275Z 16e6a4d496ed: Pull complete
2022-11-23T01:13:14.7819516Z 2133f4081ddb: Download complete
2022-11-23T01:13:14.8430172Z 3ceac802dd07: Verifying Checksum
2022-11-23T01:13:14.8430605Z 3ceac802dd07: Download complete
2022-11-23T01:13:14.9208101Z 69c929400d35: Verifying Checksum
2022-11-23T01:13:14.9208743Z 69c929400d35: Download complete
2022-11-23T01:13:15.0018699Z bc2be817cb7e: Verifying Checksum
2022-11-23T01:13:15.0019338Z bc2be817cb7e: Download complete
2022-11-23T01:13:15.7213369Z d04926f1c68d: Verifying Checksum
2022-11-23T01:13:15.7213790Z d04926f1c68d: Download complete
2022-11-23T01:13:15.7838552Z 91f116f19c0d: Verifying Checksum
2022-11-23T01:13:15.7839090Z 91f116f19c0d: Download complete
2022-11-23T01:13:15.8584414Z a7cf5b3894f8: Verifying Checksum
2022-11-23T01:13:15.8584812Z a7cf5b3894f8: Download complete
2022-11-23T01:13:23.2942648Z 3a4c0b092a9e: Download complete
2022-11-23T01:13:23.3809769Z 2e882087d824: Verifying Checksum
2022-11-23T01:13:23.3810284Z 2e882087d824: Download complete
2022-11-23T01:13:23.4643250Z ba6235196410: Download complete
2022-11-23T01:13:23.5432219Z 313c5ee380ab: Download complete
2022-11-23T01:13:23.6137476Z 6ff0fc00b0a9: Download complete
2022-11-23T01:13:23.6817789Z 06f043acdbfb: Verifying Checksum
2022-11-23T01:13:23.6818233Z 06f043acdbfb: Download complete
2022-11-23T01:13:23.7543418Z a8c562f6a1cf: Verifying Checksum
2022-11-23T01:13:23.7543832Z a8c562f6a1cf: Download complete
2022-11-23T01:13:24.6941283Z 4468cd4f574c: Verifying Checksum
2022-11-23T01:13:24.6941725Z 4468cd4f574c: Download complete
2022-11-23T01:13:24.7859277Z 46e5d3cec398: Verifying Checksum
2022-11-23T01:13:24.7859743Z 46e5d3cec398: Download complete
2022-11-23T01:13:24.8644407Z 165006759af3: Download complete
2022-11-23T01:13:24.9429388Z 2c660101e145: Verifying Checksum
2022-11-23T01:13:24.9430086Z 2c660101e145: Download complete
2022-11-23T01:13:25.0223362Z 8ce731125a4e: Download complete
2022-11-23T01:13:25.1101675Z 102ddcd90753: Verifying Checksum
2022-11-23T01:13:25.1102287Z 102ddcd90753: Download complete
2022-11-23T01:13:26.6380115Z 8859cc6b3ab7: Verifying Checksum
2022-11-23T01:13:26.6380861Z 8859cc6b3ab7: Download complete
2022-11-23T01:13:26.7203094Z 867a6ccb577f: Verifying Checksum
2022-11-23T01:13:26.7203767Z 867a6ccb577f: Download complete
2022-11-23T01:13:26.8007072Z 863c35620b44: Verifying Checksum
2022-11-23T01:13:26.8008471Z 863c35620b44: Download complete
2022-11-23T01:13:27.0936125Z 0cff5716a932: Verifying Checksum
2022-11-23T01:13:27.0937029Z 0cff5716a932: Download complete
2022-11-23T01:13:27.1714437Z f0fe61569b0b: Verifying Checksum
2022-11-23T01:13:27.1714851Z f0fe61569b0b: Download complete
2022-11-23T01:13:27.1960986Z f828f00c6d66: Verifying Checksum
2022-11-23T01:13:27.1961356Z f828f00c6d66: Download complete
2022-11-23T01:13:27.2434436Z a9f3d4742233: Verifying Checksum
2022-11-23T01:13:27.2434806Z a9f3d4742233: Download complete
2022-11-23T01:13:27.3310957Z 023a41fa48e6: Verifying Checksum
2022-11-23T01:13:27.3311377Z 023a41fa48e6: Download complete
2022-11-23T01:13:27.4508738Z 000b6751ea6f: Verifying Checksum
2022-11-23T01:13:27.4509365Z 000b6751ea6f: Download complete
2022-11-23T01:13:27.5272503Z 5970defc1d8b: Verifying Checksum
2022-11-23T01:13:27.5272932Z 5970defc1d8b: Download complete
2022-11-23T01:13:27.6000189Z 7e2d6313145f: Verifying Checksum
2022-11-23T01:13:27.6000852Z 7e2d6313145f: Download complete
2022-11-23T01:13:27.7849780Z 081025f05026: Verifying Checksum
2022-11-23T01:13:27.7850225Z 081025f05026: Download complete
2022-11-23T01:13:27.8553145Z 75f1ead35ace: Verifying Checksum
2022-11-23T01:13:27.9250119Z 793c37004dab: Verifying Checksum
2022-11-23T01:13:27.9250878Z 793c37004dab: Download complete
2022-11-23T01:13:27.9984458Z 4f1313d71da0: Download complete
2022-11-23T01:13:28.0969353Z 6386b2adbe28: Verifying Checksum
2022-11-23T01:13:28.0969779Z 6386b2adbe28: Download complete
2022-11-23T01:13:28.2790340Z f21c39caec7a: Verifying Checksum
2022-11-23T01:13:28.2790912Z f21c39caec7a: Download complete
2022-11-23T01:13:28.3593180Z 2baaae32e3d5: Verifying Checksum
2022-11-23T01:13:28.3593709Z 2baaae32e3d5: Download complete
2022-11-23T01:13:28.9551212Z 128ebe6811c5: Verifying Checksum
2022-11-23T01:13:28.9551940Z 128ebe6811c5: Download complete
2022-11-23T01:13:29.0295910Z 772fa4efddc3: Verifying Checksum
2022-11-23T01:13:29.0296448Z 772fa4efddc3: Download complete
2022-11-23T01:13:32.2480489Z 4b4d66451d67: Verifying Checksum
2022-11-23T01:13:32.2480936Z 4b4d66451d67: Download complete
2022-11-23T01:13:32.3397977Z 85cc9b957510: Verifying Checksum
2022-11-23T01:13:32.3398391Z 85cc9b957510: Download complete
2022-11-23T01:13:32.4184801Z 9ab0826d88b5: Download complete
2022-11-23T01:13:35.6527284Z 722cddd823f9: Verifying Checksum
2022-11-23T01:13:35.6527798Z 722cddd823f9: Download complete
2022-11-23T01:13:36.8736614Z 3a4c0b092a9e: Pull complete
2022-11-23T01:13:37.0135985Z d9e19734e968: Pull complete
2022-11-23T01:13:52.0086615Z c36ac12376c5: Verifying Checksum
2022-11-23T01:13:52.0087331Z c36ac12376c5: Download complete
2022-11-23T01:13:58.8735293Z 8859cc6b3ab7: Pull complete
2022-11-23T01:14:00.7587142Z 1a4b44db2103: Pull complete
2022-11-23T01:14:02.6388777Z fac00e927cfe: Pull complete
2022-11-23T01:14:10.4986947Z 2133f4081ddb: Pull complete
2022-11-23T01:14:12.4576513Z 3ceac802dd07: Pull complete
2022-11-23T01:14:14.4661892Z 69c929400d35: Pull complete
2022-11-23T01:14:16.6541177Z bc2be817cb7e: Pull complete
2022-11-23T01:14:21.5046927Z d04926f1c68d: Pull complete
2022-11-23T01:14:23.3532945Z 91f116f19c0d: Pull complete
2022-11-23T01:14:25.5375364Z a7cf5b3894f8: Pull complete
2022-11-23T01:15:01.4873084Z 722cddd823f9: Pull complete
2022-11-23T01:15:01.5999800Z 2e882087d824: Pull complete
2022-11-23T01:15:01.7032906Z ba6235196410: Pull complete
2022-11-23T01:15:01.8133540Z 313c5ee380ab: Pull complete
2022-11-23T01:15:01.9248377Z 6ff0fc00b0a9: Pull complete
2022-11-23T01:15:02.0523332Z 06f043acdbfb: Pull complete
2022-11-23T01:15:02.1925126Z a8c562f6a1cf: Pull complete
2022-11-23T01:15:04.5443306Z 4468cd4f574c: Pull complete
2022-11-23T01:15:04.6749695Z 46e5d3cec398: Pull complete
2022-11-23T01:15:04.7790446Z 165006759af3: Pull complete
2022-11-23T01:15:05.3566743Z 2c660101e145: Pull complete
2022-11-23T01:15:05.4943227Z 8ce731125a4e: Pull complete
2022-11-23T01:15:05.6432181Z 102ddcd90753: Pull complete
2022-11-23T01:15:13.8117320Z 0cff5716a932: Pull complete
2022-11-23T01:15:15.4778616Z 867a6ccb577f: Pull complete
2022-11-23T01:15:16.9649261Z 863c35620b44: Pull complete
2022-11-23T01:15:20.0365102Z f828f00c6d66: Pull complete
2022-11-23T01:15:21.8338625Z f0fe61569b0b: Pull complete
2022-11-23T01:15:24.9685425Z a9f3d4742233: Pull complete
2022-11-23T01:15:28.2616710Z 000b6751ea6f: Pull complete
2022-11-23T01:15:31.9886233Z 023a41fa48e6: Pull complete
2022-11-23T01:15:36.0432130Z 081025f05026: Pull complete
2022-11-23T01:15:38.9967548Z 5970defc1d8b: Pull complete
2022-11-23T01:15:41.0368439Z 7e2d6313145f: Pull complete
2022-11-23T01:15:49.9581449Z 4b4d66451d67: Pull complete
2022-11-23T01:15:51.9221958Z 75f1ead35ace: Pull complete
2022-11-23T01:15:53.7999663Z 793c37004dab: Pull complete
2022-11-23T01:15:55.6781302Z 4f1313d71da0: Pull complete
2022-11-23T01:15:57.0218618Z 6386b2adbe28: Pull complete
2022-11-23T01:15:58.0401853Z f21c39caec7a: Pull complete
2022-11-23T01:15:58.1671266Z 2baaae32e3d5: Pull complete
2022-11-23T01:16:00.0735677Z 128ebe6811c5: Pull complete
2022-11-23T01:16:00.2008078Z 772fa4efddc3: Pull complete
2022-11-23T01:16:28.7597044Z c36ac12376c5: Pull complete
2022-11-23T01:16:29.3026134Z 85cc9b957510: Pull complete
2022-11-23T01:16:29.3980964Z 9ab0826d88b5: Pull complete
2022-11-23T01:16:29.4109002Z Digest: sha256:a44ece0129de4f14f08fcb1423a34c97f2f88e2969bfc5ad6f33b15b4dfcfea3
2022-11-23T01:16:29.4139342Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:16:29.4182864Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:16:29.4282566Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2022-11-23T01:16:29.4282965Z with:
2022-11-23T01:16:29.4283279Z   driver-version: 515.76
2022-11-23T01:16:29.4283589Z env:
2022-11-23T01:16:29.4283882Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:16:29.4284211Z ##[endgroup]
2022-11-23T01:16:29.4320354Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-11-23T01:16:29.4320732Z with:
2022-11-23T01:16:29.4321043Z   timeout_minutes: 10
2022-11-23T01:16:29.4321373Z   max_attempts: 3
2022-11-23T01:16:29.4328039Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"
YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x

        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms

        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            sudo yum groupinstall -y "Development Tools"
            # ensure our kernel install is the same as our underlying kernel,
            # groupinstall "Development Tools" has a habit of mismatching kernel headers
            sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
            sudo modprobe backlight
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi

        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e
            nvidia-smi
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-11-23T01:16:29.4334299Z   retry_wait_seconds: 10
2022-11-23T01:16:29.4334656Z   polling_interval_seconds: 1
2022-11-23T01:16:29.4335005Z   warning_on_retry: true
2022-11-23T01:16:29.4335328Z   continue_on_error: false
2022-11-23T01:16:29.4335645Z env:
2022-11-23T01:16:29.4335959Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:16:29.4336285Z   DRIVER_VERSION: 515.76
2022-11-23T01:16:29.4336612Z ##[endgroup]
2022-11-23T01:16:29.4910176Z 
2022-11-23T01:16:29.4932105Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:16:29.4976469Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.76.run ==
2022-11-23T01:16:29.4978846Z + sudo yum remove -y nvidia-driver-latest-dkms
2022-11-23T01:16:30.0460966Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:16:30.1054518Z No Match for argument: nvidia-driver-latest-dkms
2022-11-23T01:16:30.1430599Z No Packages marked for removal
2022-11-23T01:16:30.1588259Z + echo 'Before installing NVIDIA driver'
2022-11-23T01:16:30.1588635Z + lspci
2022-11-23T01:16:30.1588959Z Before installing NVIDIA driver
2022-11-23T01:16:30.1796312Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2022-11-23T01:16:30.1796809Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2022-11-23T01:16:30.1797288Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2022-11-23T01:16:30.1797975Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2022-11-23T01:16:30.1798394Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2022-11-23T01:16:30.1799513Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2022-11-23T01:16:30.1800127Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:16:30.1800642Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:16:30.1801117Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2022-11-23T01:16:30.1801503Z + lsmod
2022-11-23T01:16:30.1819761Z Module                  Size  Used by
2022-11-23T01:16:30.1820429Z nvidia_modeset       1142784  0
2022-11-23T01:16:30.1820798Z nvidia_uvm           1269760  0
2022-11-23T01:16:30.1821118Z veth                   16384  0
2022-11-23T01:16:30.1821461Z nvidia              40808448  2 nvidia_uvm,nvidia_modeset
2022-11-23T01:16:30.1821828Z drm                   425984  1 nvidia
2022-11-23T01:16:30.1822182Z i2c_core               77824  2 nvidia,drm
2022-11-23T01:16:30.1822530Z backlight              16384  1 nvidia_modeset
2022-11-23T01:16:30.1822878Z xt_conntrack           16384  1
2022-11-23T01:16:30.1823304Z ipt_MASQUERADE         16384  1
2022-11-23T01:16:30.1823676Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2022-11-23T01:16:30.1824054Z nf_conntrack_netlink    49152  0
2022-11-23T01:16:30.1824423Z nfnetlink              16384  2 nf_conntrack_netlink
2022-11-23T01:16:30.1824787Z xfrm_user              45056  1
2022-11-23T01:16:30.1825090Z xfrm_algo              16384  1 xfrm_user
2022-11-23T01:16:30.1825430Z xt_addrtype            16384  2
2022-11-23T01:16:30.1825773Z iptable_filter         16384  1
2022-11-23T01:16:30.1826087Z iptable_nat            16384  1
2022-11-23T01:16:30.1826420Z nf_conntrack_ipv4      16384  3
2022-11-23T01:16:30.1826793Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2022-11-23T01:16:30.1827152Z nf_nat_ipv4            16384  1 iptable_nat
2022-11-23T01:16:30.1827574Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2022-11-23T01:16:30.1828110Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2022-11-23T01:16:30.1828580Z br_netfilter           24576  0
2022-11-23T01:16:30.1829085Z bridge                172032  1 br_netfilter
2022-11-23T01:16:30.1829677Z stp                    16384  1 bridge
2022-11-23T01:16:30.1830264Z llc                    16384  2 bridge,stp
2022-11-23T01:16:30.1830829Z overlay                86016  0
2022-11-23T01:16:30.1831395Z sunrpc                393216  1
2022-11-23T01:16:30.1831951Z dm_mirror              28672  0
2022-11-23T01:16:30.1832329Z dm_region_hash         20480  1 dm_mirror
2022-11-23T01:16:30.1832710Z dm_log                 20480  2 dm_region_hash,dm_mirror
2022-11-23T01:16:30.1833087Z dm_mod                143360  2 dm_log,dm_mirror
2022-11-23T01:16:30.1833409Z dax                    69632  1 dm_mod
2022-11-23T01:16:30.1833747Z sb_edac                24576  0
2022-11-23T01:16:30.1834117Z crc32_pclmul           16384  0
2022-11-23T01:16:30.1834699Z ghash_clmulni_intel    16384  0
2022-11-23T01:16:30.1835279Z pcbc                   16384  0
2022-11-23T01:16:30.1835839Z ata_piix               36864  0
2022-11-23T01:16:30.1836178Z aesni_intel           188416  0
2022-11-23T01:16:30.1836499Z libata                266240  1 ata_piix
2022-11-23T01:16:30.1837063Z aes_x86_64             20480  1 aesni_intel
2022-11-23T01:16:30.1837693Z crypto_simd            16384  1 aesni_intel
2022-11-23T01:16:30.1838253Z glue_helper            16384  1 aesni_intel
2022-11-23T01:16:30.1838600Z pcc_cpufreq            16384  0
2022-11-23T01:16:30.1839000Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2022-11-23T01:16:30.1839371Z mousedev               24576  0
2022-11-23T01:16:30.1839699Z scsi_mod              245760  1 libata
2022-11-23T01:16:30.1840102Z psmouse                32768  0
2022-11-23T01:16:30.1840896Z evdev                  20480  3
2022-11-23T01:16:30.1841455Z button                 16384  0
2022-11-23T01:16:30.1841784Z ena                   114688  0
2022-11-23T01:16:30.1842083Z xen_blkfront           49152  2
2022-11-23T01:16:30.1842414Z crc32c_intel           24576  0
2022-11-23T01:16:30.1842863Z autofs4                49152  2
2022-11-23T01:16:30.1843195Z + modinfo nvidia
2022-11-23T01:16:30.1843936Z filename:       /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2022-11-23T01:16:30.1844608Z firmware:       nvidia/515.76/gsp.bin
2022-11-23T01:16:30.1845297Z alias:          char-major-195-*
2022-11-23T01:16:30.1846425Z version:        515.76
2022-11-23T01:16:30.1847003Z supported:      external
2022-11-23T01:16:30.1847554Z license:        NVIDIA
2022-11-23T01:16:30.1848216Z srcversion:     51FD9DD90150B35351AFFBB
2022-11-23T01:16:30.1848876Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2022-11-23T01:16:30.1849265Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2022-11-23T01:16:30.1849647Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2022-11-23T01:16:30.1850048Z depends:        i2c-core,drm
2022-11-23T01:16:30.1850380Z retpoline:      Y
2022-11-23T01:16:30.1850682Z name:           nvidia
2022-11-23T01:16:30.1851130Z vermagic:       4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 
2022-11-23T01:16:30.1851579Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2022-11-23T01:16:30.1852041Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2022-11-23T01:16:30.1852456Z parm:           NVreg_ResmanDebugLevel:int
2022-11-23T01:16:30.1852817Z parm:           NVreg_RmLogonRC:int
2022-11-23T01:16:30.1853186Z parm:           NVreg_ModifyDeviceFiles:int
2022-11-23T01:16:30.1853537Z parm:           NVreg_DeviceFileUID:int
2022-11-23T01:16:30.1853900Z parm:           NVreg_DeviceFileGID:int
2022-11-23T01:16:30.1854262Z parm:           NVreg_DeviceFileMode:int
2022-11-23T01:16:30.1854677Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2022-11-23T01:16:30.1855125Z parm:           NVreg_UsePageAttributeTable:int
2022-11-23T01:16:30.1855510Z parm:           NVreg_EnablePCIeGen3:int
2022-11-23T01:16:30.1855870Z parm:           NVreg_EnableMSI:int
2022-11-23T01:16:30.1856207Z parm:           NVreg_TCEBypassMode:int
2022-11-23T01:16:30.1856588Z parm:           NVreg_EnableStreamMemOPs:int
2022-11-23T01:16:30.1857011Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2022-11-23T01:16:30.1857452Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2022-11-23T01:16:30.1857891Z parm:           NVreg_EnableS0ixPowerManagement:int
2022-11-23T01:16:30.1858361Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2022-11-23T01:16:30.1858806Z parm:           NVreg_DynamicPowerManagement:int
2022-11-23T01:16:30.1859283Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2022-11-23T01:16:30.1859746Z parm:           NVreg_EnableGpuFirmware:int
2022-11-23T01:16:30.1860135Z parm:           NVreg_EnableGpuFirmwareLogs:int
2022-11-23T01:16:30.1860560Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2022-11-23T01:16:30.1860992Z parm:           NVreg_EnableUserNUMAManagement:int
2022-11-23T01:16:30.1861385Z parm:           NVreg_MemoryPoolSize:int
2022-11-23T01:16:30.1861750Z parm:           NVreg_KMallocHeapMaxSize:int
2022-11-23T01:16:30.1862140Z parm:           NVreg_VMallocHeapMaxSize:int
2022-11-23T01:16:30.1862515Z parm:           NVreg_IgnoreMMIOCheck:int
2022-11-23T01:16:30.1862861Z parm:           NVreg_NvLinkDisable:int
2022-11-23T01:16:30.1863271Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2022-11-23T01:16:30.1863687Z parm:           NVreg_RegisterPCIDriver:int
2022-11-23T01:16:30.1864060Z parm:           NVreg_EnableDbgBreakpoint:int
2022-11-23T01:16:30.1864439Z parm:           NVreg_RegistryDwords:charp
2022-11-23T01:16:30.1864835Z parm:           NVreg_RegistryDwordsPerDevice:charp
2022-11-23T01:16:30.1865193Z parm:           NVreg_RmMsg:charp
2022-11-23T01:16:30.1865711Z parm:           NVreg_GpuBlacklist:charp
2022-11-23T01:16:30.1866093Z parm:           NVreg_TemporaryFilePath:charp
2022-11-23T01:16:30.1866455Z parm:           NVreg_ExcludedGpus:charp
2022-11-23T01:16:30.1866847Z parm:           NVreg_DmaRemapPeerMmio:int
2022-11-23T01:16:30.1867215Z parm:           rm_firmware_active:charp
2022-11-23T01:16:30.1867632Z + HAS_NVIDIA_DRIVER=0
2022-11-23T01:16:30.1868008Z ++ command -v nvidia-smi
2022-11-23T01:16:30.1868386Z + '[' -x /usr/bin/nvidia-smi ']'
2022-11-23T01:16:30.1868701Z + set +e
2022-11-23T01:16:30.1869143Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2022-11-23T01:16:34.4457937Z + INSTALLED_DRIVER_VERSION=515.76
2022-11-23T01:16:34.4458637Z + NVIDIA_SMI_STATUS=0
2022-11-23T01:16:34.4459515Z + '[' 0 -ne 0 ']'
2022-11-23T01:16:34.4460217Z + '[' 515.76 '!=' 515.76 ']'
2022-11-23T01:16:34.4460623Z + HAS_NVIDIA_DRIVER=1
2022-11-23T01:16:34.4461177Z + echo 'NVIDIA driver (515.76) has already been installed. Skipping NVIDIA driver installation'
2022-11-23T01:16:34.4461638Z + set -e
2022-11-23T01:16:34.4461962Z + '[' 1 -eq 0 ']'
2022-11-23T01:16:34.4462285Z + sudo modprobe nvidia
2022-11-23T01:16:34.4462730Z NVIDIA driver (515.76) has already been installed. Skipping NVIDIA driver installation
2022-11-23T01:16:34.4603384Z + echo 'After installing NVIDIA driver'
2022-11-23T01:16:34.4604201Z After installing NVIDIA driver
2022-11-23T01:16:34.4604845Z + lspci
2022-11-23T01:16:34.4784361Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2022-11-23T01:16:34.4785369Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2022-11-23T01:16:34.4785991Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2022-11-23T01:16:34.4786463Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2022-11-23T01:16:34.4786897Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2022-11-23T01:16:34.4787334Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2022-11-23T01:16:34.4787833Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:16:34.4788332Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:16:34.4788805Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2022-11-23T01:16:34.4789206Z + lsmod
2022-11-23T01:16:34.4806662Z Module                  Size  Used by
2022-11-23T01:16:34.4807188Z nvidia_modeset       1142784  0
2022-11-23T01:16:34.4807796Z nvidia_uvm           1269760  0
2022-11-23T01:16:34.4808390Z veth                   16384  0
2022-11-23T01:16:34.4809117Z nvidia              40808448  2 nvidia_uvm,nvidia_modeset
2022-11-23T01:16:34.4809720Z drm                   425984  1 nvidia
2022-11-23T01:16:34.4810076Z i2c_core               77824  2 nvidia,drm
2022-11-23T01:16:34.4810436Z backlight              16384  1 nvidia_modeset
2022-11-23T01:16:34.4810762Z xt_conntrack           16384  1
2022-11-23T01:16:34.4811126Z ipt_MASQUERADE         16384  1
2022-11-23T01:16:34.4811493Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2022-11-23T01:16:34.4811861Z nf_conntrack_netlink    49152  0
2022-11-23T01:16:34.4812206Z nfnetlink              16384  2 nf_conntrack_netlink
2022-11-23T01:16:34.4812550Z xfrm_user              45056  1
2022-11-23T01:16:34.4812887Z xfrm_algo              16384  1 xfrm_user
2022-11-23T01:16:34.4813209Z xt_addrtype            16384  2
2022-11-23T01:16:34.4813544Z iptable_filter         16384  1
2022-11-23T01:16:34.4813877Z iptable_nat            16384  1
2022-11-23T01:16:34.4814196Z nf_conntrack_ipv4      16384  3
2022-11-23T01:16:34.4814561Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2022-11-23T01:16:34.4814929Z nf_nat_ipv4            16384  1 iptable_nat
2022-11-23T01:16:34.4815301Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2022-11-23T01:16:34.4815840Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2022-11-23T01:16:34.4816552Z br_netfilter           24576  0
2022-11-23T01:16:34.4816898Z bridge                172032  1 br_netfilter
2022-11-23T01:16:34.4817220Z stp                    16384  1 bridge
2022-11-23T01:16:34.4817560Z llc                    16384  2 bridge,stp
2022-11-23T01:16:34.4818036Z overlay                86016  0
2022-11-23T01:16:34.4818592Z sunrpc                393216  1
2022-11-23T01:16:34.4819120Z dm_mirror              28672  0
2022-11-23T01:16:34.4819711Z dm_region_hash         20480  1 dm_mirror
2022-11-23T01:16:34.4820142Z dm_log                 20480  2 dm_region_hash,dm_mirror
2022-11-23T01:16:34.4820514Z dm_mod                143360  2 dm_log,dm_mirror
2022-11-23T01:16:34.4820848Z dax                    69632  1 dm_mod
2022-11-23T01:16:34.4821148Z sb_edac                24576  0
2022-11-23T01:16:34.4821465Z crc32_pclmul           16384  0
2022-11-23T01:16:34.4821789Z ghash_clmulni_intel    16384  0
2022-11-23T01:16:34.4822108Z pcbc                   16384  0
2022-11-23T01:16:34.4822424Z ata_piix               36864  0
2022-11-23T01:16:34.4822747Z aesni_intel           188416  0
2022-11-23T01:16:34.4823055Z libata                266240  1 ata_piix
2022-11-23T01:16:34.4823393Z aes_x86_64             20480  1 aesni_intel
2022-11-23T01:16:34.4823742Z crypto_simd            16384  1 aesni_intel
2022-11-23T01:16:34.4824098Z glue_helper            16384  1 aesni_intel
2022-11-23T01:16:34.4824413Z pcc_cpufreq            16384  0
2022-11-23T01:16:34.4824804Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2022-11-23T01:16:34.4825189Z mousedev               24576  0
2022-11-23T01:16:34.4825497Z scsi_mod              245760  1 libata
2022-11-23T01:16:34.4825816Z psmouse                32768  0
2022-11-23T01:16:34.4826126Z evdev                  20480  3
2022-11-23T01:16:34.4826417Z button                 16384  0
2022-11-23T01:16:34.4826716Z ena                   114688  0
2022-11-23T01:16:34.4827026Z xen_blkfront           49152  2
2022-11-23T01:16:34.4827333Z crc32c_intel           24576  0
2022-11-23T01:16:34.4827644Z autofs4                49152  2
2022-11-23T01:16:34.4827957Z + modinfo nvidia
2022-11-23T01:16:34.4828523Z filename:       /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2022-11-23T01:16:34.4828925Z firmware:       nvidia/515.76/gsp.bin
2022-11-23T01:16:34.4829329Z alias:          char-major-195-*
2022-11-23T01:16:34.4829662Z version:        515.76
2022-11-23T01:16:34.4829960Z supported:      external
2022-11-23T01:16:34.4830271Z license:        NVIDIA
2022-11-23T01:16:34.4830610Z srcversion:     51FD9DD90150B35351AFFBB
2022-11-23T01:16:34.4830966Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2022-11-23T01:16:34.4831340Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2022-11-23T01:16:34.4831708Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2022-11-23T01:16:34.4832088Z depends:        i2c-core,drm
2022-11-23T01:16:34.4832420Z retpoline:      Y
2022-11-23T01:16:34.4832728Z name:           nvidia
2022-11-23T01:16:34.4833176Z vermagic:       4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 
2022-11-23T01:16:34.4833620Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2022-11-23T01:16:34.4834080Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2022-11-23T01:16:34.4834512Z parm:           NVreg_ResmanDebugLevel:int
2022-11-23T01:16:34.4834854Z parm:           NVreg_RmLogonRC:int
2022-11-23T01:16:34.4835217Z parm:           NVreg_ModifyDeviceFiles:int
2022-11-23T01:16:34.4835583Z parm:           NVreg_DeviceFileUID:int
2022-11-23T01:16:34.4835930Z parm:           NVreg_DeviceFileGID:int
2022-11-23T01:16:34.4836299Z parm:           NVreg_DeviceFileMode:int
2022-11-23T01:16:34.4836724Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2022-11-23T01:16:34.4837146Z parm:           NVreg_UsePageAttributeTable:int
2022-11-23T01:16:34.4837529Z parm:           NVreg_EnablePCIeGen3:int
2022-11-23T01:16:34.4837885Z parm:           NVreg_EnableMSI:int
2022-11-23T01:16:34.4838319Z parm:           NVreg_TCEBypassMode:int
2022-11-23T01:16:34.4838698Z parm:           NVreg_EnableStreamMemOPs:int
2022-11-23T01:16:34.4839119Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2022-11-23T01:16:34.4839558Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2022-11-23T01:16:34.4840051Z parm:           NVreg_EnableS0ixPowerManagement:int
2022-11-23T01:16:34.4840533Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2022-11-23T01:16:34.4840993Z parm:           NVreg_DynamicPowerManagement:int
2022-11-23T01:16:34.4841458Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2022-11-23T01:16:34.4841918Z parm:           NVreg_EnableGpuFirmware:int
2022-11-23T01:16:34.4842316Z parm:           NVreg_EnableGpuFirmwareLogs:int
2022-11-23T01:16:34.4842720Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2022-11-23T01:16:34.4843152Z parm:           NVreg_EnableUserNUMAManagement:int
2022-11-23T01:16:34.4843553Z parm:           NVreg_MemoryPoolSize:int
2022-11-23T01:16:34.4843920Z parm:           NVreg_KMallocHeapMaxSize:int
2022-11-23T01:16:34.4844312Z parm:           NVreg_VMallocHeapMaxSize:int
2022-11-23T01:16:34.4844683Z parm:           NVreg_IgnoreMMIOCheck:int
2022-11-23T01:16:34.4845030Z parm:           NVreg_NvLinkDisable:int
2022-11-23T01:16:34.4845446Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2022-11-23T01:16:34.4846232Z parm:           NVreg_RegisterPCIDriver:int
2022-11-23T01:16:34.4846613Z parm:           NVreg_EnableDbgBreakpoint:int
2022-11-23T01:16:34.4847216Z parm:           NVreg_RegistryDwords:charp
2022-11-23T01:16:34.4847630Z parm:           NVreg_RegistryDwordsPerDevice:charp
2022-11-23T01:16:34.4848075Z parm:           NVreg_RmMsg:charp
2022-11-23T01:16:34.4848471Z parm:           NVreg_GpuBlacklist:charp
2022-11-23T01:16:34.4848950Z parm:           NVreg_TemporaryFilePath:charp
2022-11-23T01:16:34.4849342Z parm:           NVreg_ExcludedGpus:charp
2022-11-23T01:16:34.4849778Z parm:           NVreg_DmaRemapPeerMmio:int
2022-11-23T01:16:34.4850195Z parm:           rm_firmware_active:charp
2022-11-23T01:16:34.4850541Z + set +e
2022-11-23T01:16:34.4850983Z + nvidia-smi
2022-11-23T01:16:38.2760491Z Wed Nov 23 01:16:38 2022       
2022-11-23T01:16:38.2761244Z +-----------------------------------------------------------------------------+
2022-11-23T01:16:38.2761876Z | NVIDIA-SMI 515.76       Driver Version: 515.76       CUDA Version: 11.7     |
2022-11-23T01:16:38.2762607Z |-------------------------------+----------------------+----------------------+
2022-11-23T01:16:38.2763293Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-11-23T01:16:38.2763876Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-11-23T01:16:38.2764350Z |                               |                      |               MIG M. |
2022-11-23T01:16:38.2764779Z |===============================+======================+======================|
2022-11-23T01:16:38.2819466Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |           8589873520 |
2022-11-23T01:16:38.2819923Z | N/A   33C    P0    39W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-11-23T01:16:38.2820357Z |                               |                      |                  N/A |
2022-11-23T01:16:38.2820943Z +-------------------------------+----------------------+----------------------+
2022-11-23T01:16:38.2873284Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |          12884871352 |
2022-11-23T01:16:38.2873719Z | N/A   28C    P0    39W / 150W |      0MiB /  7680MiB |     99%      Default |
2022-11-23T01:16:38.2874172Z |                               |                      |                  N/A |
2022-11-23T01:16:38.2874770Z +-------------------------------+----------------------+----------------------+
2022-11-23T01:16:38.2875235Z                                                                                
2022-11-23T01:16:38.2875852Z +-----------------------------------------------------------------------------+
2022-11-23T01:16:38.2876597Z | Processes:                                                                  |
2022-11-23T01:16:38.2877071Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-11-23T01:16:38.2877615Z |        ID   ID                                                   Usage      |
2022-11-23T01:16:38.2878099Z |=============================================================================|
2022-11-23T01:16:38.2878911Z |  No running processes found                                                 |
2022-11-23T01:16:38.2879937Z +-----------------------------------------------------------------------------+
2022-11-23T01:16:38.3454218Z + NVIDIA_SMI_STATUS=0
2022-11-23T01:16:38.3455022Z + '[' 0 -eq 0 ']'
2022-11-23T01:16:38.3455468Z + echo 'INFO: Ignoring allowed status 0'
2022-11-23T01:16:38.3455921Z + set -e
2022-11-23T01:16:38.3456329Z INFO: Ignoring allowed status 0
2022-11-23T01:16:38.3459791Z == Installing nvidia container toolkit for amzn2 ==
2022-11-23T01:16:38.3464052Z + sudo yum install -y yum-utils
2022-11-23T01:16:38.9001635Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:16:40.2937762Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-11-23T01:16:40.2938332Z Nothing to do
2022-11-23T01:16:40.3779713Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-11-23T01:16:40.9455471Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:16:40.9874389Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-11-23T01:16:40.9875256Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-11-23T01:16:40.9875966Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-11-23T01:16:41.0028199Z + sudo yum install -y nvidia-docker2
2022-11-23T01:16:41.5597666Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:16:42.9108921Z Package nvidia-docker2-2.11.0-1.noarch already installed and latest version
2022-11-23T01:16:42.9109506Z Nothing to do
2022-11-23T01:16:42.9911052Z + sudo systemctl restart docker
2022-11-23T01:17:04.5320772Z Command completed after 1 attempt(s).
2022-11-23T01:17:04.5321055Z 
2022-11-23T01:17:04.5324459Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:17:04.5377156Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-11-23T01:17:04.5377629Z python3 -m pip install psutil==5.9.1
2022-11-23T01:17:04.5378020Z python3 -m pip install pynvml==11.4.1
2022-11-23T01:17:04.5378454Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-11-23T01:17:04.5378913Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2022-11-23T01:17:04.5392223Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:17:04.5392577Z env:
2022-11-23T01:17:04.5392897Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:04.5393244Z   GPU_FLAG: --gpus all
2022-11-23T01:17:04.5393545Z ##[endgroup]
2022-11-23T01:17:04.8366879Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:17:04.8599881Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1)
2022-11-23T01:17:05.4403648Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:17:05.4630688Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1)
2022-11-23T01:17:05.7556962Z Prepare all required actions
2022-11-23T01:17:05.7557440Z Getting action download info
2022-11-23T01:17:05.9149358Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2022-11-23T01:17:06.1798572Z Download action repository 'actions/download-artifact@v3' (SHA:9782bd6a9848b53b110e712e20e42d89988822b7)
2022-11-23T01:17:06.3302583Z ##[group]Run ./.github/actions/download-build-artifacts
2022-11-23T01:17:06.3302953Z with:
2022-11-23T01:17:06.3303285Z   name: linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T01:17:06.3303622Z env:
2022-11-23T01:17:06.3303922Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:06.3304259Z   GPU_FLAG: --gpus all
2022-11-23T01:17:06.3304556Z ##[endgroup]
2022-11-23T01:17:06.3338139Z ##[group]Run seemethere/download-artifact-s3@v4
2022-11-23T01:17:06.3338491Z with:
2022-11-23T01:17:06.3338845Z   name: linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T01:17:06.3339229Z   s3-bucket: gha-artifacts
2022-11-23T01:17:06.3339619Z   region: us-east-1
2022-11-23T01:17:06.3339915Z env:
2022-11-23T01:17:06.3340225Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:06.3340571Z   GPU_FLAG: --gpus all
2022-11-23T01:17:06.3340879Z ##[endgroup]
2022-11-23T01:17:06.9106073Z Found 1 objects with prefix pytorch/pytorch/3528293554/linux-bionic-cuda11.7-py3.10-gcc7/
2022-11-23T01:17:06.9106815Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-11-23T01:17:13.2362099Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-11-23T01:17:13.2362722Z 
2022-11-23T01:17:13.2368077Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:17:13.2369479Z Artifact download has finished successfully
2022-11-23T01:17:13.2655883Z ##[group]Run unzip -o artifacts.zip
2022-11-23T01:17:13.2656277Z unzip -o artifacts.zip
2022-11-23T01:17:13.2670020Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:17:13.2670362Z env:
2022-11-23T01:17:13.2670674Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:13.2671028Z   GPU_FLAG: --gpus all
2022-11-23T01:17:13.2671337Z ##[endgroup]
2022-11-23T01:17:13.2715829Z Archive:  artifacts.zip
2022-11-23T01:17:13.2717871Z    creating: dist/
2022-11-23T01:17:15.3366205Z   inflating: dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl  
2022-11-23T01:17:15.3367093Z    creating: build/custom_test_artifacts/
2022-11-23T01:17:15.3368009Z    creating: build/custom_test_artifacts/custom-op-build/
2022-11-23T01:17:15.3368897Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-11-23T01:17:15.3375262Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:17:15.3376475Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-11-23T01:17:15.3377721Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:17:15.3378948Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:17:15.3380049Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:17:15.3381387Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:17:15.3383428Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:17:15.3384666Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:17:15.3385911Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:17:15.3388970Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:17:15.3390867Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:17:15.3392977Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:17:15.3394523Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:17:15.3396570Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:17:15.3398307Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:17:15.3399554Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:17:15.3400782Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:17:15.3454719Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:17:15.3456283Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:17:15.3457818Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:17:15.3459398Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:17:15.3460965Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:17:15.3462451Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:17:15.3463901Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:17:15.3465435Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:17:15.3466909Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:17:15.3506874Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:17:15.3548668Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:17:15.3550222Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:17:15.3551635Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:17:15.3552958Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:17:15.3554334Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:17:15.3555702Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:17:15.3557059Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:17:15.3558828Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:17:15.3633011Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:17:15.3706938Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:17:15.3708342Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:17:15.3709534Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-11-23T01:17:15.3710878Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-11-23T01:17:15.3712063Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:17:15.3713397Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-11-23T01:17:15.3714642Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-11-23T01:17:15.3715958Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-11-23T01:17:15.3717236Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-11-23T01:17:15.3718484Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-11-23T01:17:15.3719755Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-11-23T01:17:15.3721062Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-11-23T01:17:15.3722317Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-11-23T01:17:15.3723507Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-11-23T01:17:15.3724633Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-11-23T01:17:15.3744062Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-11-23T01:17:15.3857396Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-11-23T01:17:15.3858647Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-11-23T01:17:15.3859931Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-11-23T01:17:15.3861313Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-11-23T01:17:15.3862634Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-11-23T01:17:15.3863918Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-11-23T01:17:15.3865248Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-11-23T01:17:15.3866529Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-11-23T01:17:15.3867870Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-11-23T01:17:15.3869106Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-11-23T01:17:15.3870268Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-11-23T01:17:15.3889782Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-11-23T01:17:15.3972191Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-11-23T01:17:15.3973605Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:17:15.3974897Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:17:15.3976173Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-11-23T01:17:15.3977266Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-11-23T01:17:15.3978427Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:17:15.3979630Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-11-23T01:17:15.3983240Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-11-23T01:17:15.3984351Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-11-23T01:17:15.3985474Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-11-23T01:17:15.4078124Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-11-23T01:17:15.4141970Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-11-23T01:17:15.4143018Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-11-23T01:17:15.4143990Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-11-23T01:17:15.4151258Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:17:15.4152385Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-11-23T01:17:15.4153587Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:17:15.4154762Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:17:15.4155860Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:17:15.4157192Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:17:15.4159086Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:17:15.4160320Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:17:15.4161529Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:17:15.4164485Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:17:15.4166448Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:17:15.4168834Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:17:15.4170199Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:17:15.4172293Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:17:15.4173933Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:17:15.4175216Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:17:15.4176496Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:17:15.4231593Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:17:15.4233136Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:17:15.4234707Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:17:15.4236258Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:17:15.4237806Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:17:15.4239256Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:17:15.4240731Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:17:15.4242232Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:17:15.4243876Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:17:15.4284643Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:17:15.4327462Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:17:15.4329364Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:17:15.4330775Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:17:15.4332113Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:17:15.4333469Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:17:15.4334846Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:17:15.4336177Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:17:15.4338007Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:17:15.4413179Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:17:15.4487680Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:17:15.4489054Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:17:15.4490221Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-11-23T01:17:15.4491361Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-11-23T01:17:15.4492541Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:17:15.4493720Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-11-23T01:17:15.4494992Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-11-23T01:17:15.4496335Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-11-23T01:17:15.4497613Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-11-23T01:17:15.4498809Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-11-23T01:17:15.4500057Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-11-23T01:17:15.4501304Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-11-23T01:17:15.4502647Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-11-23T01:17:15.4503905Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-11-23T01:17:15.4505081Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-11-23T01:17:15.4525391Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-11-23T01:17:15.4590352Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-11-23T01:17:15.4591743Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:17:15.4593016Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:17:15.4594207Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-11-23T01:17:15.4595447Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-11-23T01:17:15.4596618Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:17:15.4597740Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-11-23T01:17:15.4600870Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-11-23T01:17:15.4602114Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-11-23T01:17:15.4602934Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-11-23T01:17:15.4652606Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-11-23T01:17:15.4653174Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-11-23T01:17:15.4653744Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-11-23T01:17:15.4660552Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:17:15.4661214Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-11-23T01:17:15.4661870Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:17:15.4662544Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:17:15.4663195Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:17:15.4664619Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:17:15.4666461Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:17:15.4667132Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:17:15.4667779Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:17:15.4669924Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:17:15.4671034Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:17:15.4672663Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:17:15.4673753Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:17:15.4675679Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:17:15.4676512Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:17:15.4677215Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:17:15.4677882Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:17:15.4732299Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:17:15.4733113Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:17:15.4733954Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:17:15.4734799Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:17:15.4735628Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:17:15.4736426Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:17:15.4737388Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:17:15.4738211Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:17:15.4739131Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:17:15.4780782Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:17:15.4822113Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:17:15.4823204Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:17:15.4824185Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:17:15.4824936Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:17:15.4825915Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:17:15.4826807Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:17:15.4827671Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:17:15.4829923Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:17:15.4903657Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:17:15.4977121Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:17:15.4977852Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:17:15.4978521Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-11-23T01:17:15.4979348Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-11-23T01:17:15.4979994Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:17:15.4980629Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-11-23T01:17:15.4981466Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-11-23T01:17:15.4982333Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-11-23T01:17:15.4983207Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-11-23T01:17:15.4984091Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-11-23T01:17:15.4985049Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-11-23T01:17:15.4986177Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-11-23T01:17:15.4986917Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-11-23T01:17:15.4987965Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-11-23T01:17:15.4988685Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-11-23T01:17:15.4993700Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-11-23T01:17:15.5141496Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-11-23T01:17:15.5142223Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-11-23T01:17:15.5143032Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-11-23T01:17:15.5143781Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-11-23T01:17:15.5144506Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-11-23T01:17:15.5145203Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-11-23T01:17:15.5145931Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-11-23T01:17:15.5146855Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-11-23T01:17:15.5147750Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-11-23T01:17:15.5148793Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-11-23T01:17:15.5149507Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-11-23T01:17:15.5171015Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-11-23T01:17:15.5229576Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-11-23T01:17:15.5230332Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:17:15.5231032Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:17:15.5231704Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-11-23T01:17:15.5232673Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-11-23T01:17:15.5234010Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:17:15.5234776Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-11-23T01:17:15.5237783Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-11-23T01:17:15.5238586Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-11-23T01:17:15.5239440Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-11-23T01:17:15.5358799Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-11-23T01:17:15.5404464Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-11-23T01:17:15.5404909Z    creating: build/lib/
2022-11-23T01:17:15.5405651Z   inflating: build/lib/libclog.a     
2022-11-23T01:17:15.5472144Z   inflating: build/lib/libgtest.a    
2022-11-23T01:17:15.5482453Z   inflating: build/lib/libpthreadpool.a  
2022-11-23T01:17:15.5588872Z   inflating: build/lib/libprotobuf-lite.a  
2022-11-23T01:17:15.5680220Z   inflating: build/lib/libbenchmark.a  
2022-11-23T01:17:15.5689804Z   inflating: build/lib/libittnotify.a  
2022-11-23T01:17:15.5721652Z   inflating: build/lib/libtensorpipe_uv.a  
2022-11-23T01:17:15.5798592Z   inflating: build/lib/libasmjit.a   
2022-11-23T01:17:15.6332714Z   inflating: build/lib/libprotobuf.a  
2022-11-23T01:17:15.6465630Z   inflating: build/lib/libgloo.a     
2022-11-23T01:17:15.6498176Z   inflating: build/lib/libfmt.a      
2022-11-23T01:17:15.6500116Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-11-23T01:17:15.6500785Z   inflating: build/lib/libfoxi_loader.a  
2022-11-23T01:17:15.6580591Z   inflating: build/lib/libc10.so     
2022-11-23T01:17:15.6581782Z   inflating: build/lib/libtorch_global_deps.so  
2022-11-23T01:17:15.6591900Z   inflating: build/lib/libcpuinfo.a  
2022-11-23T01:17:15.6600792Z   inflating: build/lib/libcpuinfo_internals.a  
2022-11-23T01:17:15.6603456Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-11-23T01:17:15.7175617Z   inflating: build/lib/libprotoc.a   
2022-11-23T01:17:15.7194386Z   inflating: build/lib/libgmock.a    
2022-11-23T01:17:15.7195194Z   inflating: build/lib/libgtest_main.a  
2022-11-23T01:17:15.7196209Z   inflating: build/lib/libbenchmark_main.a  
2022-11-23T01:17:15.7338627Z   inflating: build/lib/libXNNPACK.a  
2022-11-23T01:17:16.7121957Z   inflating: build/lib/libdnnl.a     
2022-11-23T01:17:16.7777044Z   inflating: build/lib/libtensorpipe.a  
2022-11-23T01:17:16.7830155Z   inflating: build/lib/libc10_cuda.so  
2022-11-23T01:17:16.7846052Z   inflating: build/lib/libqnnpack.a  
2022-11-23T01:17:16.7847998Z   inflating: build/lib/libgmock_main.a  
2022-11-23T01:17:16.9390919Z   inflating: build/lib/libfbgemm.a   
2022-11-23T01:17:16.9414004Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-11-23T01:17:17.0560572Z   inflating: build/lib/libdnnl_graph.a  
2022-11-23T01:17:17.1076777Z   inflating: build/lib/libkineto.a   
2022-11-23T01:17:17.1366607Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-11-23T01:17:17.1412129Z   inflating: build/lib/libcaffe2_protos.a  
2022-11-23T01:17:17.1460195Z   inflating: build/lib/libonnx_proto.a  
2022-11-23T01:17:17.1482304Z   inflating: build/lib/libnnpack.a   
2022-11-23T01:17:17.2160837Z   inflating: build/lib/libonnx.a     
2022-11-23T01:17:17.2594970Z   inflating: build/lib/libgloo_cuda.a  
2022-11-23T01:17:19.6243868Z   inflating: build/lib/libtorch_cpu.so  
2022-11-23T01:17:21.7494735Z   inflating: build/lib/libtorch_cuda.so  
2022-11-23T01:17:21.7495600Z   inflating: build/lib/libtorch.so   
2022-11-23T01:17:21.7498864Z   inflating: build/lib/libc10d_cuda_test.so  
2022-11-23T01:17:22.7391462Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-11-23T01:17:22.7415547Z   inflating: build/lib/libjitbackend_test.so  
2022-11-23T01:17:22.7475566Z   inflating: build/lib/libtorchbind_test.so  
2022-11-23T01:17:22.7506563Z   inflating: build/lib/libbackend_with_compiler.so  
2022-11-23T01:17:22.7511434Z   inflating: build/lib/libshm.so     
2022-11-23T01:17:22.9315703Z   inflating: build/lib/libtorch_python.so  
2022-11-23T01:17:22.9355397Z   inflating: build/lib/libnnapi_backend.so  
2022-11-23T01:17:22.9355786Z    creating: build/bin/
2022-11-23T01:17:22.9408730Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-11-23T01:17:22.9463961Z   inflating: build/bin/c10_DeviceGuard_test  
2022-11-23T01:17:22.9517643Z   inflating: build/bin/c10_Device_test  
2022-11-23T01:17:22.9569038Z   inflating: build/bin/c10_StreamGuard_test  
2022-11-23T01:17:22.9632031Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-11-23T01:17:22.9684642Z   inflating: build/bin/c10_SymInt_test  
2022-11-23T01:17:22.9744180Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-11-23T01:17:22.9803886Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-11-23T01:17:22.9865130Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-11-23T01:17:22.9916487Z   inflating: build/bin/c10_Array_test  
2022-11-23T01:17:22.9973350Z   inflating: build/bin/c10_Bitset_test  
2022-11-23T01:17:23.0028187Z   inflating: build/bin/c10_C++17_test  
2022-11-23T01:17:23.0079342Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-11-23T01:17:23.0132109Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-11-23T01:17:23.0184972Z   inflating: build/bin/c10_Half_test  
2022-11-23T01:17:23.0245506Z   inflating: build/bin/c10_LeftRight_test  
2022-11-23T01:17:23.0312811Z   inflating: build/bin/c10_Metaprogramming_test  
2022-11-23T01:17:23.0468706Z   inflating: build/bin/c10_SmallVectorTest  
2022-11-23T01:17:23.0522407Z   inflating: build/bin/c10_Synchronized_test  
2022-11-23T01:17:23.0583753Z   inflating: build/bin/c10_ThreadLocal_test  
2022-11-23T01:17:23.0639988Z   inflating: build/bin/c10_TypeIndex_test  
2022-11-23T01:17:23.0693832Z   inflating: build/bin/c10_TypeList_test  
2022-11-23T01:17:23.0745128Z   inflating: build/bin/c10_TypeTraits_test  
2022-11-23T01:17:23.0800427Z   inflating: build/bin/c10_accumulate_test  
2022-11-23T01:17:23.0860167Z   inflating: build/bin/c10_bfloat16_test  
2022-11-23T01:17:23.0917843Z   inflating: build/bin/c10_complex_math_test  
2022-11-23T01:17:23.0977422Z   inflating: build/bin/c10_complex_test  
2022-11-23T01:17:23.1095384Z   inflating: build/bin/c10_either_test  
2022-11-23T01:17:23.1151390Z   inflating: build/bin/c10_exception_test  
2022-11-23T01:17:23.1204796Z   inflating: build/bin/c10_flags_test  
2022-11-23T01:17:23.1387997Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-11-23T01:17:23.1441718Z   inflating: build/bin/c10_irange_test  
2022-11-23T01:17:23.1503678Z   inflating: build/bin/c10_logging_test  
2022-11-23T01:17:23.1583645Z   inflating: build/bin/c10_optional_test  
2022-11-23T01:17:23.1649946Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-11-23T01:17:23.1709028Z   inflating: build/bin/c10_registry_test  
2022-11-23T01:17:23.1772560Z   inflating: build/bin/c10_string_view_test  
2022-11-23T01:17:23.1827667Z   inflating: build/bin/c10_tempfile_test  
2022-11-23T01:17:23.1888020Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-11-23T01:17:23.1948811Z   inflating: build/bin/c10_typeid_test  
2022-11-23T01:17:23.2470922Z   inflating: build/bin/protoc-3.13.0.0  
2022-11-23T01:17:23.2992583Z   inflating: build/bin/protoc        
2022-11-23T01:17:23.3044455Z   inflating: build/bin/c10_cuda_CUDATest  
2022-11-23T01:17:23.3360757Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-11-23T01:17:23.3716712Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-11-23T01:17:23.3781506Z   inflating: build/bin/TCPStoreTest  
2022-11-23T01:17:23.3838894Z   inflating: build/bin/HashStoreTest  
2022-11-23T01:17:23.3896429Z   inflating: build/bin/FileStoreTest  
2022-11-23T01:17:23.3912316Z   inflating: build/bin/ProcessGroupMPITest  
2022-11-23T01:17:23.3915588Z   inflating: build/bin/example_allreduce  
2022-11-23T01:17:23.3971861Z   inflating: build/bin/Dimname_test  
2022-11-23T01:17:23.4050172Z   inflating: build/bin/Dict_test     
2022-11-23T01:17:23.4118874Z   inflating: build/bin/MaybeOwned_test  
2022-11-23T01:17:23.4179771Z   inflating: build/bin/NamedTensor_test  
2022-11-23T01:17:23.4242915Z   inflating: build/bin/apply_utils_test  
2022-11-23T01:17:23.4306071Z   inflating: build/bin/atest         
2022-11-23T01:17:23.4372062Z   inflating: build/bin/basic         
2022-11-23T01:17:23.4429662Z   inflating: build/bin/broadcast_test  
2022-11-23T01:17:23.4492176Z   inflating: build/bin/cpu_generator_test  
2022-11-23T01:17:23.4548140Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-11-23T01:17:23.4601319Z   inflating: build/bin/dispatch_key_set_test  
2022-11-23T01:17:23.4696651Z   inflating: build/bin/cpu_rng_test  
2022-11-23T01:17:23.4749666Z   inflating: build/bin/dlconvertor_test  
2022-11-23T01:17:23.4812107Z   inflating: build/bin/extension_backend_test  
2022-11-23T01:17:23.4871823Z   inflating: build/bin/half_test     
2022-11-23T01:17:23.4973698Z   inflating: build/bin/ivalue_test   
2022-11-23T01:17:23.5026334Z   inflating: build/bin/lazy_tensor_test  
2022-11-23T01:17:23.5083782Z   inflating: build/bin/memory_format_test  
2022-11-23T01:17:23.5141590Z   inflating: build/bin/math_kernel_test  
2022-11-23T01:17:23.5198638Z   inflating: build/bin/memory_overlapping_test  
2022-11-23T01:17:23.5259124Z   inflating: build/bin/native_test   
2022-11-23T01:17:23.5314858Z   inflating: build/bin/mobile_memory_cleanup  
2022-11-23T01:17:23.5369338Z   inflating: build/bin/operator_name_test  
2022-11-23T01:17:23.5422789Z   inflating: build/bin/operators_test  
2022-11-23T01:17:23.5478878Z   inflating: build/bin/packedtensoraccessor_test  
2022-11-23T01:17:23.5549171Z   inflating: build/bin/pow_test      
2022-11-23T01:17:23.5610830Z   inflating: build/bin/quantized_test  
2022-11-23T01:17:23.5663664Z   inflating: build/bin/reduce_ops_test  
2022-11-23T01:17:23.5717677Z   inflating: build/bin/reportMemoryUsage_test  
2022-11-23T01:17:23.5778376Z   inflating: build/bin/scalar_tensor_test  
2022-11-23T01:17:23.5839676Z   inflating: build/bin/scalar_test   
2022-11-23T01:17:23.5895510Z   inflating: build/bin/stride_properties_test  
2022-11-23T01:17:23.5980382Z   inflating: build/bin/tensor_iterator_test  
2022-11-23T01:17:23.5983134Z   inflating: build/bin/thread_init_test  
2022-11-23T01:17:23.6042754Z   inflating: build/bin/type_ptr_test  
2022-11-23T01:17:23.6102727Z   inflating: build/bin/test_parallel  
2022-11-23T01:17:23.6155566Z   inflating: build/bin/variant_test  
2022-11-23T01:17:23.6220711Z   inflating: build/bin/type_test     
2022-11-23T01:17:23.6276438Z   inflating: build/bin/undefined_tensor_test  
2022-11-23T01:17:23.6277716Z   inflating: build/bin/verify_api_visibility  
2022-11-23T01:17:23.6352802Z   inflating: build/bin/vmap_test     
2022-11-23T01:17:23.6407534Z   inflating: build/bin/weakref_test  
2022-11-23T01:17:23.6461863Z   inflating: build/bin/wrapdim_test  
2022-11-23T01:17:23.6525921Z   inflating: build/bin/IListRef_test  
2022-11-23T01:17:23.6644441Z   inflating: build/bin/List_test     
2022-11-23T01:17:23.6696890Z   inflating: build/bin/xla_tensor_test  
2022-11-23T01:17:23.6828347Z   inflating: build/bin/kernel_function_legacy_test  
2022-11-23T01:17:23.6932344Z   inflating: build/bin/kernel_function_test  
2022-11-23T01:17:23.7002640Z   inflating: build/bin/KernelFunction_test  
2022-11-23T01:17:23.7141818Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-11-23T01:17:23.7253367Z   inflating: build/bin/kernel_lambda_test  
2022-11-23T01:17:23.7317487Z   inflating: build/bin/kernel_stackbased_test  
2022-11-23T01:17:23.7420776Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-11-23T01:17:23.7474625Z   inflating: build/bin/CppSignature_test  
2022-11-23T01:17:23.7525913Z   inflating: build/bin/op_allowlist_test  
2022-11-23T01:17:23.7582453Z   inflating: build/bin/inline_container_test  
2022-11-23T01:17:23.7642765Z   inflating: build/bin/backend_fallback_test  
2022-11-23T01:17:23.7957062Z   inflating: build/bin/op_registration_test  
2022-11-23T01:17:23.8013124Z   inflating: build/bin/cuda_apply_test  
2022-11-23T01:17:23.8077175Z   inflating: build/bin/cuda_atomic_ops_test  
2022-11-23T01:17:23.8134864Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-11-23T01:17:23.8207333Z   inflating: build/bin/cuda_complex_math_test  
2022-11-23T01:17:23.8259720Z   inflating: build/bin/cuda_device_test  
2022-11-23T01:17:23.8322151Z   inflating: build/bin/cuda_complex_test  
2022-11-23T01:17:23.8385536Z   inflating: build/bin/cuda_cub_test  
2022-11-23T01:17:23.8438820Z   inflating: build/bin/cuda_dlconvertor_test  
2022-11-23T01:17:23.8492756Z   inflating: build/bin/cuda_integer_divider_test  
2022-11-23T01:17:23.8564582Z   inflating: build/bin/cuda_distributions_test  
2022-11-23T01:17:23.8627690Z   inflating: build/bin/cuda_generator_test  
2022-11-23T01:17:23.8680297Z   inflating: build/bin/cuda_half_test  
2022-11-23T01:17:23.8732671Z   inflating: build/bin/cuda_optional_test  
2022-11-23T01:17:23.8788733Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-11-23T01:17:23.8853584Z   inflating: build/bin/cuda_stream_test  
2022-11-23T01:17:23.8908460Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-11-23T01:17:23.8960439Z   inflating: build/bin/cuda_cudnn_test  
2022-11-23T01:17:23.9016377Z   inflating: build/bin/cuda_vectorized_test  
2022-11-23T01:17:23.9034040Z   inflating: build/bin/tutorial_tensorexpr  
2022-11-23T01:17:23.9104334Z   inflating: build/bin/ProcessGroupGlooTest  
2022-11-23T01:17:23.9166809Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-11-23T01:17:23.9232788Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-11-23T01:17:23.9295747Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-11-23T01:17:23.9352362Z   inflating: build/bin/ProcessGroupUCCTest  
2022-11-23T01:17:23.9410361Z   inflating: build/bin/test_dist_autograd  
2022-11-23T01:17:23.9485336Z   inflating: build/bin/test_cpp_rpc  
2022-11-23T01:17:23.9488300Z   inflating: build/bin/parallel_benchmark  
2022-11-23T01:17:23.9562238Z   inflating: build/bin/test_mobile_nnc  
2022-11-23T01:17:23.9573813Z   inflating: build/bin/aot_model_compiler_test  
2022-11-23T01:17:24.0489034Z   inflating: build/bin/test_tensorexpr  
2022-11-23T01:17:24.0876048Z   inflating: build/bin/test_lazy     
2022-11-23T01:17:24.0881660Z   inflating: build/bin/torch_shm_manager  
2022-11-23T01:17:24.2209586Z   inflating: build/bin/test_api      
2022-11-23T01:17:24.3419822Z   inflating: build/bin/test_jit      
2022-11-23T01:17:24.3423273Z   inflating: .pytorch-test-times.json  
2022-11-23T01:17:24.3454240Z ##[group]Run df -H
2022-11-23T01:17:24.3454572Z df -H
2022-11-23T01:17:24.3467950Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:17:24.3468308Z env:
2022-11-23T01:17:24.3468627Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:24.3468981Z   GPU_FLAG: --gpus all
2022-11-23T01:17:24.3469289Z ##[endgroup]
2022-11-23T01:17:24.3509242Z Filesystem      Size  Used Avail Use% Mounted on
2022-11-23T01:17:24.3509663Z devtmpfs        129G     0  129G   0% /dev
2022-11-23T01:17:24.3510038Z tmpfs           129G  6.3M  129G   1% /dev/shm
2022-11-23T01:17:24.3510627Z tmpfs           129G  553k  129G   1% /run
2022-11-23T01:17:24.3510991Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2022-11-23T01:17:24.3511357Z /dev/xvda1      162G   28G  134G  18% /
2022-11-23T01:17:24.3535880Z ##[group]Run .github/scripts/parse_ref.py
2022-11-23T01:17:24.3536300Z .github/scripts/parse_ref.py
2022-11-23T01:17:24.3547901Z shell: /usr/bin/bash -e {0}
2022-11-23T01:17:24.3548229Z env:
2022-11-23T01:17:24.3548529Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:24.3548876Z   GPU_FLAG: --gpus all
2022-11-23T01:17:24.3549198Z ##[endgroup]
2022-11-23T01:17:24.3841246Z ##[group]Run set -x
2022-11-23T01:17:24.3841683Z set -x
2022-11-23T01:17:24.3841978Z 
2022-11-23T01:17:24.3842294Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-11-23T01:17:24.3842714Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-11-23T01:17:24.3843148Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-11-23T01:17:24.3843523Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-11-23T01:17:24.3843867Z else
2022-11-23T01:17:24.3844215Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-11-23T01:17:24.3844557Z fi
2022-11-23T01:17:24.3844826Z 
2022-11-23T01:17:24.3845208Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-11-23T01:17:24.3845598Z 
2022-11-23T01:17:24.3846298Z # sanitize the input commit message and PR body here:
2022-11-23T01:17:24.3846662Z #
2022-11-23T01:17:24.3847121Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-11-23T01:17:24.3847684Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-11-23T01:17:24.3848173Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-11-23T01:17:24.3848551Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-11-23T01:17:24.3848864Z 
2022-11-23T01:17:24.3849279Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-11-23T01:17:24.3849723Z # wreak havoc internally
2022-11-23T01:17:24.3850111Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-11-23T01:17:24.3850490Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-11-23T01:17:24.3850827Z 
2022-11-23T01:17:24.3851209Z # detached container should get cleaned up by teardown_ec2_linux
2022-11-23T01:17:24.3851663Z # TODO: Stop building test binaries as part of the build phase
2022-11-23T01:17:24.3852110Z # Used for GPU_FLAG since that doesn't play nice
2022-11-23T01:17:24.3852504Z # shellcheck disable=SC2086,SC2090
2022-11-23T01:17:24.3852875Z container_name=$(docker run \
2022-11-23T01:17:24.3853202Z   ${GPU_FLAG:-} \
2022-11-23T01:17:24.3853539Z   -e BUILD_ENVIRONMENT \
2022-11-23T01:17:24.3854016Z   -e PR_NUMBER \
2022-11-23T01:17:24.3854337Z   -e GITHUB_ACTIONS \
2022-11-23T01:17:24.3854667Z   -e BASE_SHA \
2022-11-23T01:17:24.3854983Z   -e BRANCH \
2022-11-23T01:17:24.3855282Z   -e SHA1 \
2022-11-23T01:17:24.3855610Z   -e AWS_DEFAULT_REGION \
2022-11-23T01:17:24.3855945Z   -e IN_WHEEL_TEST \
2022-11-23T01:17:24.3856257Z   -e SHARD_NUMBER \
2022-11-23T01:17:24.3856577Z   -e TEST_CONFIG \
2022-11-23T01:17:24.3856900Z   -e NUM_TEST_SHARDS \
2022-11-23T01:17:24.3857209Z   -e PR_BODY \
2022-11-23T01:17:24.3857535Z   -e COMMIT_MESSAGES \
2022-11-23T01:17:24.3857895Z   -e PYTORCH_RETRY_TEST_CASES \
2022-11-23T01:17:24.3858257Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-11-23T01:17:24.3858607Z   -e PR_LABELS \
2022-11-23T01:17:24.3858962Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-11-23T01:17:24.3859328Z   -e SCCACHE_BUCKET \
2022-11-23T01:17:24.3859659Z   -e SCCACHE_S3_KEY_PREFIX \
2022-11-23T01:17:24.3859989Z   -e XLA_CUDA \
2022-11-23T01:17:24.3860344Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-11-23T01:17:24.3860718Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2022-11-23T01:17:24.3861113Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2022-11-23T01:17:24.3861535Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-11-23T01:17:24.3861911Z   --ulimit stack=10485760:83886080 \
2022-11-23T01:17:24.3862289Z   --security-opt seccomp=unconfined \
2022-11-23T01:17:24.3862758Z   --cap-add=SYS_PTRACE \
2022-11-23T01:17:24.3863102Z   --ipc=host \
2022-11-23T01:17:24.3863419Z   --shm-size="${SHM_SIZE}" \
2022-11-23T01:17:24.3863756Z   --tty \
2022-11-23T01:17:24.3864057Z   --detach \
2022-11-23T01:17:24.3864378Z   --name="${container_name}" \
2022-11-23T01:17:24.3864724Z   --user jenkins \
2022-11-23T01:17:24.3865112Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-11-23T01:17:24.3865501Z   -w /var/lib/jenkins/workspace \
2022-11-23T01:17:24.3865843Z   "${DOCKER_IMAGE}"
2022-11-23T01:17:24.3866144Z )
2022-11-23T01:17:24.3866495Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2022-11-23T01:17:24.3867014Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2022-11-23T01:17:24.3879338Z shell: /usr/bin/bash -e {0}
2022-11-23T01:17:24.3879647Z env:
2022-11-23T01:17:24.3879957Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:17:24.3880279Z   GPU_FLAG: --gpus all
2022-11-23T01:17:24.3880668Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T01:17:24.3881039Z   PR_NUMBER: 
2022-11-23T01:17:24.3881330Z   BRANCH: master
2022-11-23T01:17:24.3881670Z   SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:17:24.3882061Z   BASE_SHA: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:17:24.3882407Z   PYTORCH_RETRY_TEST_CASES: 1
2022-11-23T01:17:24.3882751Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-11-23T01:17:24.3883088Z   TEST_CONFIG: distributed
2022-11-23T01:17:24.3883411Z   SHARD_NUMBER: 1
2022-11-23T01:17:24.3883690Z   NUM_TEST_SHARDS: 3
2022-11-23T01:17:24.3883988Z   PR_BODY: 
2022-11-23T01:17:24.3884359Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-11-23T01:17:24.3884731Z   SCCACHE_S3_KEY_PREFIX: trunk
2022-11-23T01:17:24.3885050Z   SHM_SIZE: 2g
2022-11-23T01:17:24.3885610Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:17:24.3886371Z   XLA_CUDA: 
2022-11-23T01:17:24.3886786Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-11-23T01:17:24.3887229Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2022-11-23T01:17:24.3887697Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2022-11-23T01:17:24.3888036Z ##[endgroup]
2022-11-23T01:17:24.3917133Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-11-23T01:17:24.3917709Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *onnx* ]]
2022-11-23T01:17:24.3918119Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-11-23T01:17:24.3920955Z ++ git cherry -v origin/master
2022-11-23T01:17:24.3938147Z + COMMIT_MESSAGES=
2022-11-23T01:17:24.3938477Z + COMMIT_MESSAGES=
2022-11-23T01:17:24.3938776Z + PR_BODY=
2022-11-23T01:17:24.3939091Z + export COMMIT_MESSAGES=
2022-11-23T01:17:24.3939423Z + COMMIT_MESSAGES=
2022-11-23T01:17:24.3939740Z + export PR_BODY=
2022-11-23T01:17:24.3940033Z + PR_BODY=
2022-11-23T01:17:24.3949040Z +++ nproc --ignore=2
2022-11-23T01:17:24.3961632Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3528293554 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:17:38.4255902Z + container_name=2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T01:17:38.4256564Z + echo DOCKER_CONTAINER_ID=2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T01:17:38.4261012Z ++ echo dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl
2022-11-23T01:17:38.4263131Z + docker exec -t 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f sh -c 'pip install dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2022-11-23T01:17:38.9946269Z Processing ./dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl
2022-11-23T01:17:39.9529149Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (2.6.3)
2022-11-23T01:17:39.9532799Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (4.4.0)
2022-11-23T01:17:39.9537474Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (1.11.1)
2022-11-23T01:17:39.9554851Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (3.3.0)
2022-11-23T01:17:39.9635476Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==1.14.0a0+git1cfd385) (1.21.2)
2022-11-23T01:17:39.9859296Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch==1.14.0a0+git1cfd385) (1.2.1)
2022-11-23T01:17:40.9256836Z Installing collected packages: torch
2022-11-23T01:17:50.9022726Z Successfully installed torch-1.14.0a0+git1cfd385
2022-11-23T01:17:50.9672618Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-11-23T01:17:50.9902978Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch
2022-11-23T01:17:50.9903547Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T01:17:50.9904092Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib
2022-11-23T01:17:50.9904632Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test
2022-11-23T01:17:50.9907047Z + BUILD_DIR=build
2022-11-23T01:17:50.9907423Z + BUILD_RENAMED_DIR=build_renamed
2022-11-23T01:17:50.9907787Z + BUILD_BIN_DIR=build/bin
2022-11-23T01:17:50.9908429Z + export VALGRIND=ON
2022-11-23T01:17:50.9909212Z + VALGRIND=ON
2022-11-23T01:17:50.9909743Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *clang9* ]]
2022-11-23T01:17:50.9910268Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *bazel* ]]
2022-11-23T01:17:50.9910670Z ++ realpath build/custom_test_artifacts
2022-11-23T01:17:50.9916765Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-11-23T01:17:50.9920190Z ++ dirname .jenkins/pytorch/test.sh
2022-11-23T01:17:50.9927659Z + source .jenkins/pytorch/common.sh
2022-11-23T01:17:50.9931414Z +++ dirname .jenkins/pytorch/common.sh
2022-11-23T01:17:50.9941915Z ++ source .jenkins/pytorch/common_utils.sh
2022-11-23T01:17:50.9944155Z +++ declare -f -t trap_add
2022-11-23T01:17:50.9949484Z ++ set -ex
2022-11-23T01:17:50.9949973Z ++ [[ linux-bionic-cuda11.7-py3.10-gcc7 == *rocm* ]]
2022-11-23T01:17:50.9950617Z ++ BUILD_TEST_LIBTORCH=0
2022-11-23T01:17:50.9951027Z + echo 'Environment variables'
2022-11-23T01:17:50.9951384Z Environment variables
2022-11-23T01:17:50.9951692Z + env
2022-11-23T01:17:50.9959550Z SHARD_NUMBER=1
2022-11-23T01:17:50.9959957Z NV_LIBCUBLAS_DEV_VERSION=11.10.1.25-1
2022-11-23T01:17:50.9960494Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-7
2022-11-23T01:17:50.9961118Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-11-23T01:17:50.9961618Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.13.4-1+cuda11.7
2022-11-23T01:17:50.9961986Z UCC_HOME=/usr
2022-11-23T01:17:50.9962452Z BUILD_ENVIRONMENT=linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T01:17:50.9962862Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2022-11-23T01:17:50.9963465Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-7=11.7.3.21-1
2022-11-23T01:17:50.9964262Z INSTALLED_DB=yes
2022-11-23T01:17:50.9964844Z HOSTNAME=2ed9f7ebac38
2022-11-23T01:17:50.9965386Z GITHUB_REF_NAME=master
2022-11-23T01:17:50.9966408Z GITHUB_API_URL=https://api.github.com
2022-11-23T01:17:50.9966922Z OPENSSL_DIR=/opt/openssl
2022-11-23T01:17:50.9967549Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee
2022-11-23T01:17:50.9968257Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_35013f0a-5192-4638-8268-f4d67edbdc47
2022-11-23T01:17:50.9968754Z CUDA_PATH=/usr/local/cuda
2022-11-23T01:17:50.9969332Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2022-11-23T01:17:50.9969783Z GITHUB_RUN_ATTEMPT=1
2022-11-23T01:17:50.9970124Z TEST_CONFIG=distributed
2022-11-23T01:17:50.9970510Z NV_LIBNPP_VERSION=11.7.3.21-1
2022-11-23T01:17:50.9970946Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-7=11.7.50-1
2022-11-23T01:17:50.9971349Z GITHUB_REPOSITORY_OWNER=pytorch
2022-11-23T01:17:50.9971712Z GITHUB_ACTIONS=true
2022-11-23T01:17:50.9972035Z NVIDIA_VISIBLE_DEVICES=all
2022-11-23T01:17:50.9972420Z NV_NVPROF_VERSION=11.7.50-1
2022-11-23T01:17:50.9972880Z NV_LIBCUSPARSE_VERSION=11.7.3.50-1
2022-11-23T01:17:50.9973209Z CI=true
2022-11-23T01:17:50.9973565Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2022-11-23T01:17:50.9974080Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-7=11.10.1.25-1
2022-11-23T01:17:50.9974466Z BRANCH=master
2022-11-23T01:17:50.9974796Z GITHUB_HEAD_REF=
2022-11-23T01:17:50.9975201Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2022-11-23T01:17:50.9975593Z GITHUB_ACTOR=pytorchmergebot
2022-11-23T01:17:50.9976011Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2022-11-23T01:17:50.9976405Z GITHUB_ACTION_REF=
2022-11-23T01:17:50.9976763Z NCCL_VERSION=2.13.4-1
2022-11-23T01:17:50.9977108Z GITHUB_ACTION=__self
2022-11-23T01:17:50.9977439Z VALGRIND=ON
2022-11-23T01:17:50.9977779Z GITHUB_REF_PROTECTED=true
2022-11-23T01:17:50.9978327Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2022-11-23T01:17:50.9978816Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2022-11-23T01:17:50.9982744Z ***
2022-11-23T01:17:50.9983072Z INSTALLED_VISION=yes
2022-11-23T01:17:50.9983398Z NVARCH=x86_64
2022-11-23T01:17:50.9983812Z NV_LIBCUSPARSE_DEV_VERSION=11.7.3.50-1
2022-11-23T01:17:50.9984162Z HOME=/var/lib/jenkins
2022-11-23T01:17:50.9984925Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_35013f0a-5192-4638-8268-f4d67edbdc47
2022-11-23T01:17:50.9985408Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2022-11-23T01:17:50.9985752Z GITHUB_ACTION_REPOSITORY=
2022-11-23T01:17:50.9986105Z GITHUB_REF_TYPE=branch
2022-11-23T01:17:50.9986496Z NV_LIBNCCL_PACKAGE_VERSION=2.13.4-1
2022-11-23T01:17:50.9986848Z GITHUB_RETENTION_DAYS=90
2022-11-23T01:17:50.9987316Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2022-11-23T01:17:50.9987805Z NV_LIBNCCL_PACKAGE=libnccl2=2.13.4-1+cuda11.7
2022-11-23T01:17:50.9988483Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_35013f0a-5192-4638-8268-f4d67edbdc47
2022-11-23T01:17:50.9989001Z DEBIAN_FRONTEND=noninteractive
2022-11-23T01:17:50.9989708Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2022-11-23T01:17:50.9990160Z GITHUB_REF=refs/heads/master
2022-11-23T01:17:50.9990570Z NV_CUDA_LIB_VERSION=11.7.0-1
2022-11-23T01:17:50.9991038Z GITHUB_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:17:50.9991511Z INSTALLED_PROTOBUF=yes
2022-11-23T01:17:50.9991924Z GITHUB_RUN_ID=3528293554
2022-11-23T01:17:50.9992432Z NV_LIBNPP_PACKAGE=libnpp-11-7=11.7.3.21-1
2022-11-23T01:17:50.9992902Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2022-11-23T01:17:50.9993302Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2022-11-23T01:17:50.9993768Z NV_NVTX_VERSION=11.7.50-1
2022-11-23T01:17:50.9994263Z GITHUB_SERVER_URL=https://github.com
2022-11-23T01:17:50.9994645Z MAX_JOBS=30
2022-11-23T01:17:50.9995091Z NV_LIBCUBLAS_VERSION=11.10.1.25-1
2022-11-23T01:17:50.9995624Z NV_LIBCUBLAS_PACKAGE=libcublas-11-7=11.10.1.25-1
2022-11-23T01:17:50.9996322Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2022-11-23T01:17:50.9996889Z UCX_HOME=/usr
2022-11-23T01:17:50.9997314Z PYTORCH_RETRY_TEST_CASES=1
2022-11-23T01:17:50.9997802Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2022-11-23T01:17:50.9998249Z BASE_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:17:50.9998792Z NV_CUDA_CUDART_DEV_VERSION=11.7.60-1
2022-11-23T01:17:50.9999226Z PR_BODY=
2022-11-23T01:17:50.9999557Z GITHUB_BASE_REF=
2022-11-23T01:17:50.9999944Z TERM=xterm
2022-11-23T01:17:51.0000319Z XLA_CUDA=
2022-11-23T01:17:51.0000742Z NV_NVML_DEV_VERSION=11.7.50-1
2022-11-23T01:17:51.0001166Z TORCH_CUDA_ARCH_LIST=Maxwell
2022-11-23T01:17:51.0001562Z CUDA_VERSION=11.7.0
2022-11-23T01:17:51.0001995Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-7
2022-11-23T01:17:51.0002441Z OPENSSL_ROOT_DIR=/opt/openssl
2022-11-23T01:17:51.0003152Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_35013f0a-5192-4638-8268-f4d67edbdc47
2022-11-23T01:17:51.0003630Z GITHUB_JOB=test
2022-11-23T01:17:51.0004017Z SCCACHE_S3_KEY_PREFIX=trunk
2022-11-23T01:17:51.0004420Z COMMIT_MESSAGES=
2022-11-23T01:17:51.0004802Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2022-11-23T01:17:51.0005269Z NUM_TEST_SHARDS=3
2022-11-23T01:17:51.0005635Z PR_NUMBER=
2022-11-23T01:17:51.0006713Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_35013f0a-5192-4638-8268-f4d67edbdc47
2022-11-23T01:17:51.0007247Z SHLVL=1
2022-11-23T01:17:51.0007771Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-7
2022-11-23T01:17:51.0008202Z GITHUB_REPOSITORY=pytorch/pytorch
2022-11-23T01:17:51.0009463Z NVIDIA_REQUIRE_CUDA=cuda>=11.7 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511
2022-11-23T01:17:51.0011289Z NV_LIBNPP_DEV_VERSION=11.7.3.21-1
2022-11-23T01:17:51.0011708Z SHA1=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:17:51.0012179Z GITHUB_EVENT_NAME=push
2022-11-23T01:17:51.0012648Z NV_CUDA_CUDART_VERSION=11.7.60-1
2022-11-23T01:17:51.0013099Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2022-11-23T01:17:51.0013537Z GITHUB_RUN_NUMBER=18336
2022-11-23T01:17:51.0013978Z GITHUB_WORKFLOW=trunk
2022-11-23T01:17:51.0014510Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:17:51.0015121Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.13.4-1
2022-11-23T01:17:51.0015715Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:17:51.0016237Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2022-11-23T01:17:51.0016656Z _=/usr/bin/env
2022-11-23T01:17:51.0017085Z + echo 'Testing pytorch'
2022-11-23T01:17:51.0017485Z Testing pytorch
2022-11-23T01:17:51.0017850Z + export LANG=C.UTF-8
2022-11-23T01:17:51.0018254Z + LANG=C.UTF-8
2022-11-23T01:17:51.0018665Z + PR_NUMBER=
2022-11-23T01:17:51.0019013Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-11-23T01:17:51.0019437Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-11-23T01:17:51.0068692Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *rocm* ]]
2022-11-23T01:17:51.0069103Z + [[ distributed == \s\l\o\w ]]
2022-11-23T01:17:51.0069588Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *slow-gradcheck* ]]
2022-11-23T01:17:51.0070098Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *cuda* ]]
2022-11-23T01:17:51.0070511Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-11-23T01:17:51.0071043Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-11-23T01:17:51.0071436Z + [[ distributed == *crossref* ]]
2022-11-23T01:17:51.0071781Z + [[ distributed == *dynamo* ]]
2022-11-23T01:17:51.0072137Z + [[ distributed == *inductor* ]]
2022-11-23T01:17:51.0072606Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *rocm* ]]
2022-11-23T01:17:51.0073175Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *-bazel-* ]]
2022-11-23T01:17:51.0073636Z + pip_install --user ninja==1.10.2
2022-11-23T01:17:51.0074112Z + pip install --progress-bar off --user ninja==1.10.2
2022-11-23T01:17:51.5612864Z Collecting ninja==1.10.2
2022-11-23T01:17:51.5858475Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2022-11-23T01:17:52.4544416Z Installing collected packages: ninja
2022-11-23T01:17:52.4646970Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-11-23T01:17:52.4647740Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T01:17:52.4716618Z Successfully installed ninja-1.10.2
2022-11-23T01:17:52.5375852Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:17:52.5376929Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:17:52.5378075Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *asan* ]]
2022-11-23T01:17:52.5378586Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *-tsan* ]]
2022-11-23T01:17:52.5379025Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-11-23T01:17:52.5379422Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2022-11-23T01:17:52.5385242Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *tbb* ]]
2022-11-23T01:17:52.5399584Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *libtorch* ]]
2022-11-23T01:17:52.5400137Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *-bazel-* ]]
2022-11-23T01:17:52.5400651Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *-tsan* ]]
2022-11-23T01:17:52.5403241Z + cd test
2022-11-23T01:17:52.5403729Z + python -c 'import torch; print(torch.__config__.show())'
2022-11-23T01:17:54.2309604Z PyTorch built with:
2022-11-23T01:17:54.2310405Z   - GCC 7.5
2022-11-23T01:17:54.2310804Z   - C++ Version: 201402
2022-11-23T01:17:54.2311422Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-11-23T01:17:54.2312067Z   - Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289)
2022-11-23T01:17:54.2312527Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-11-23T01:17:54.2312978Z   - LAPACK is enabled (usually provided by MKL)
2022-11-23T01:17:54.2313389Z   - NNPACK is enabled
2022-11-23T01:17:54.2313774Z   - CPU capability usage: AVX2
2022-11-23T01:17:54.2314163Z   - CUDA Runtime 11.7
2022-11-23T01:17:54.2314647Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2022-11-23T01:17:54.2315053Z   - CuDNN 8.5
2022-11-23T01:17:54.2315401Z   - Magma 2.6.1
2022-11-23T01:17:54.2318609Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=1.14.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-11-23T01:17:54.2320906Z 
2022-11-23T01:17:54.4553641Z + cd test
2022-11-23T01:17:54.4554247Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-11-23T01:17:56.0725362Z ATen/Parallel:
2022-11-23T01:17:56.0741462Z 	at::get_num_threads() : 16
2022-11-23T01:17:56.0741860Z 	at::get_num_interop_threads() : 16
2022-11-23T01:17:56.0742222Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-11-23T01:17:56.0742554Z 	omp_get_max_threads() : 16
2022-11-23T01:17:56.0743264Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-11-23T01:17:56.0743747Z 	mkl_get_max_threads() : 16
2022-11-23T01:17:56.0744233Z Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289)
2022-11-23T01:17:56.0744661Z std::thread::hardware_concurrency() : 32
2022-11-23T01:17:56.0745046Z Environment variables:
2022-11-23T01:17:56.0745365Z 	OMP_NUM_THREADS : [not set]
2022-11-23T01:17:56.0745712Z 	MKL_NUM_THREADS : [not set]
2022-11-23T01:17:56.0746052Z ATen parallel backend: OpenMP
2022-11-23T01:17:56.0746271Z 
2022-11-23T01:17:56.3034873Z + [[ distributed == *backward* ]]
2022-11-23T01:17:56.3035574Z + [[ distributed == *xla* ]]
2022-11-23T01:17:56.3036033Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-11-23T01:17:56.3036648Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *libtorch* ]]
2022-11-23T01:17:56.3037039Z + [[ distributed == distributed ]]
2022-11-23T01:17:56.3037380Z + install_filelock
2022-11-23T01:17:56.3037698Z + pip_install filelock
2022-11-23T01:17:56.3038108Z + pip install --progress-bar off filelock
2022-11-23T01:17:56.8226365Z Collecting filelock
2022-11-23T01:17:56.8425626Z   Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
2022-11-23T01:17:57.7268611Z Installing collected packages: filelock
2022-11-23T01:17:57.7658184Z Successfully installed filelock-3.8.0
2022-11-23T01:17:57.8338167Z + install_triton
2022-11-23T01:17:57.8338507Z + local commit
2022-11-23T01:17:57.8339070Z + [[ distributed == *rocm* ]]
2022-11-23T01:17:57.8342695Z ++ get_pinned_commit triton
2022-11-23T01:17:57.8343077Z ++ cat .github/ci_commit_pins/triton.txt
2022-11-23T01:17:57.8358556Z + commit=0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:17:57.8359313Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:17:57.8360103Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:17:58.3086004Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:17:58.3092309Z   Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-rwxe6u7_
2022-11-23T01:17:58.3113001Z   Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-rwxe6u7_
2022-11-23T01:17:59.0917539Z   Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331'
2022-11-23T01:17:59.0938765Z   Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:17:59.5024939Z   Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:17:59.6971082Z   Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:17:59.6972936Z   Running command git submodule update --init --recursive -q
2022-11-23T01:18:00.3223438Z   Preparing metadata (setup.py) ... [?25l- done
2022-11-23T01:18:00.5505129Z [?25hCollecting cmake
2022-11-23T01:18:00.5731368Z   Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB)
2022-11-23T01:18:01.0327166Z Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (3.8.0)
2022-11-23T01:18:01.0331908Z Requirement already satisfied: torch in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (1.14.0a0+git1cfd385)
2022-11-23T01:18:01.0590170Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (2.6.3)
2022-11-23T01:18:01.0594896Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (4.4.0)
2022-11-23T01:18:01.0599621Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (1.11.1)
2022-11-23T01:18:01.0815541Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch->triton==2.0.0) (1.2.1)
2022-11-23T01:18:01.0887510Z Building wheels for collected packages: triton
2022-11-23T01:18:53.9360404Z   Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | done
2022-11-23T01:18:53.9841571Z [?25h  Created wheel for triton: filename=triton-2.0.0-cp310-cp310-linux_x86_64.whl size=15377935 sha256=715c8c0f84179122b694e714da140f26d0f92a145c14bf634805845e63d422a6
2022-11-23T01:18:53.9843909Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/3f/1d/23/1c2bc47d618a44f9c949aea4b7e355e737a1f1ed208f009295
2022-11-23T01:18:53.9868301Z Successfully built triton
2022-11-23T01:18:54.8576677Z Installing collected packages: cmake, triton
2022-11-23T01:18:56.8762897Z Successfully installed cmake-3.25.0 triton-2.0.0
2022-11-23T01:18:56.9820084Z + pip_install --user jinja2
2022-11-23T01:18:56.9820607Z + pip install --progress-bar off --user jinja2
2022-11-23T01:18:58.0087235Z Collecting jinja2
2022-11-23T01:18:58.0404406Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2022-11-23T01:18:58.3257922Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2) (2.1.1)
2022-11-23T01:18:59.2050994Z Installing collected packages: jinja2
2022-11-23T01:18:59.3114560Z Successfully installed jinja2-3.1.2
2022-11-23T01:18:59.3781153Z + test_distributed
2022-11-23T01:18:59.3781949Z + echo 'Testing distributed python tests'
2022-11-23T01:18:59.3782329Z Testing distributed python tests
2022-11-23T01:18:59.3782837Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose
2022-11-23T01:19:01.6296508Z Ignoring disabled issues:  []
2022-11-23T01:19:01.6691031Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T01:19:01.6691674Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T01:19:01.6698602Z Found test time stats from artifacts
2022-11-23T01:19:01.6714090Z Selected tests:
2022-11-23T01:19:01.6714523Z  distributed/algorithms/quantization/test_quantization
2022-11-23T01:19:01.6714944Z  distributed/test_distributed_spawn
2022-11-23T01:19:01.6716653Z  distributed/rpc/test_faulty_agent
2022-11-23T01:19:01.6717685Z  distributed/pipeline/sync/test_stream
2022-11-23T01:19:01.6718098Z  distributed/pipeline/sync/test_phony
2022-11-23T01:19:01.6718506Z  distributed/pipeline/sync/test_dependency
2022-11-23T01:19:01.6719940Z  distributed/pipeline/sync/test_checkpoint
2022-11-23T01:19:01.6720394Z  distributed/pipeline/sync/skip/test_verify_skippables
2022-11-23T01:19:01.6720819Z  distributed/pipeline/sync/skip/test_portal
2022-11-23T01:19:01.6721219Z  distributed/pipeline/sync/skip/test_gpipe
2022-11-23T01:19:01.6721580Z  distributed/fsdp/test_shard_utils
2022-11-23T01:19:01.6721970Z  distributed/checkpoint/test_traverse
2022-11-23T01:19:01.6722369Z  distributed/_shard/test_replicated_tensor
2022-11-23T01:19:01.6722703Z  distributed/test_nccl
2022-11-23T01:19:01.6723245Z  distributed/_composable/test_contract
2022-11-23T01:19:01.6723643Z  distributed/checkpoint/test_utils
2022-11-23T01:19:01.6724007Z  distributed/elastic/utils/logging_test
2022-11-23T01:19:01.6724394Z  distributed/fsdp/test_checkpoint_wrapper
2022-11-23T01:19:01.6724763Z  distributed/test_data_parallel
2022-11-23T01:19:01.6725128Z  distributed/fsdp/test_fsdp_traversal
2022-11-23T01:19:01.6725486Z  distributed/fsdp/test_fsdp_uneven
2022-11-23T01:19:01.6726198Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-11-23T01:19:01.6726621Z  distributed/_shard/sharded_tensor/ops/test_embedding
2022-11-23T01:19:01.6727024Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-11-23T01:19:01.6727409Z  distributed/test_c10d_error_logger
2022-11-23T01:19:01.6727807Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-11-23T01:19:01.6728220Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T01:19:01.6728632Z  distributed/_tensor/parallel/test_parallelize_api
2022-11-23T01:19:01.6729027Z  distributed/_tensor/test_math_ops
2022-11-23T01:19:01.6731913Z  distributed/_shard/sharded_tensor/ops/test_tensor_ops
2022-11-23T01:19:01.6732679Z  distributed/_shard/sharding_plan/test_sharding_plan
2022-11-23T01:19:01.6733496Z  distributed/fsdp/test_fsdp_memory
2022-11-23T01:19:01.6734286Z  distributed/fsdp/test_fsdp_tp_integration
2022-11-23T01:19:01.6734762Z  distributed/_composable/test_fully_shard
2022-11-23T01:19:01.6735160Z  distributed/test_c10d_spawn_ucc
2022-11-23T01:19:01.6735543Z  distributed/fsdp/test_fsdp_clip_grad_norm
2022-11-23T01:19:01.6735947Z  distributed/elastic/multiprocessing/api_test
2022-11-23T01:19:01.6736628Z  distributed/_tensor/test_matrix_ops
2022-11-23T01:19:01.6737154Z  distributed/fsdp/test_fsdp_exec_order
2022-11-23T01:19:01.6737510Z  distributed/test_c10d_spawn_gloo
2022-11-23T01:19:01.6737860Z  distributed/test_c10d_spawn_nccl
2022-11-23T01:19:01.6738400Z  distributed/fsdp/test_fsdp_grad_acc
2022-11-23T01:19:01.6739103Z  distributed/test_store
2022-11-23T01:19:01.6739703Z  distributed/fsdp/test_wrap
2022-11-23T01:19:01.6740102Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T01:19:01.6740529Z  distributed/fsdp/test_fsdp_use_orig_params
2022-11-23T01:19:01.6740907Z  distributed/fsdp/test_fsdp_mixed_precision
2022-11-23T01:19:01.6741302Z  distributed/rpc/cuda/test_tensorpipe_agent
2022-11-23T01:19:01.6741690Z Prioritized test from test file changes.
2022-11-23T01:19:01.6742228Z reordering tests for PR:
2022-11-23T01:19:01.6742558Z prioritized: []
2022-11-23T01:19:01.6747098Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/rpc/test_faulty_agent', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/fsdp/test_shard_utils', 'distributed/checkpoint/test_traverse', 'distributed/_shard/test_replicated_tensor', 'distributed/test_nccl', 'distributed/_composable/test_contract', 'distributed/checkpoint/test_utils', 'distributed/elastic/utils/logging_test', 'distributed/fsdp/test_checkpoint_wrapper', 'distributed/test_data_parallel', 'distributed/fsdp/test_fsdp_traversal', 'distributed/fsdp/test_fsdp_uneven', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/_shard/sharded_tensor/ops/test_embedding', 'distributed/_shard/sharded_tensor/ops/test_chunk', 'distributed/test_c10d_error_logger', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/_shard/sharded_tensor/ops/test_linear', 'distributed/_tensor/parallel/test_parallelize_api', 'distributed/_tensor/test_math_ops', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/_shard/sharding_plan/test_sharding_plan', 'distributed/fsdp/test_fsdp_memory', 'distributed/fsdp/test_fsdp_tp_integration', 'distributed/_composable/test_fully_shard', 'distributed/test_c10d_spawn_ucc', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/elastic/multiprocessing/api_test', 'distributed/_tensor/test_matrix_ops', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/test_c10d_spawn_gloo', 'distributed/test_c10d_spawn_nccl', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/test_store', 'distributed/fsdp/test_wrap', 'distributed/_shard/sharded_tensor/test_sharded_tensor', 'distributed/fsdp/test_fsdp_use_orig_params', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/rpc/cuda/test_tensorpipe_agent']
2022-11-23T01:19:01.6750199Z 
2022-11-23T01:19:01.6750778Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T01:19:01.7569090Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T01:19:01.7751590Z parallel (file granularity) tests:
2022-11-23T01:19:01.7752197Z  
2022-11-23T01:19:01.7752682Z serial (file granularity) tests:
2022-11-23T01:19:01.7753534Z  distributed/algorithms/quantization/test_quantization
2022-11-23T01:19:01.7754091Z  distributed/test_distributed_spawn
2022-11-23T01:19:01.7754442Z  distributed/rpc/test_faulty_agent
2022-11-23T01:19:01.7754819Z  distributed/pipeline/sync/test_stream
2022-11-23T01:19:01.7755206Z  distributed/pipeline/sync/test_phony
2022-11-23T01:19:01.7755586Z  distributed/pipeline/sync/test_dependency
2022-11-23T01:19:01.7755986Z  distributed/pipeline/sync/test_checkpoint
2022-11-23T01:19:01.7756405Z  distributed/pipeline/sync/skip/test_verify_skippables
2022-11-23T01:19:01.7756810Z  distributed/pipeline/sync/skip/test_portal
2022-11-23T01:19:01.7757243Z  distributed/pipeline/sync/skip/test_gpipe
2022-11-23T01:19:01.7757622Z  distributed/fsdp/test_shard_utils
2022-11-23T01:19:01.7757990Z  distributed/checkpoint/test_traverse
2022-11-23T01:19:01.7758360Z  distributed/_shard/test_replicated_tensor
2022-11-23T01:19:01.7758732Z  distributed/test_nccl
2022-11-23T01:19:01.7759080Z  distributed/_composable/test_contract
2022-11-23T01:19:01.7759434Z  distributed/checkpoint/test_utils
2022-11-23T01:19:01.7759807Z  distributed/elastic/utils/logging_test
2022-11-23T01:19:01.7760196Z  distributed/fsdp/test_checkpoint_wrapper
2022-11-23T01:19:01.7760709Z  distributed/test_data_parallel
2022-11-23T01:19:01.7761084Z  distributed/fsdp/test_fsdp_traversal
2022-11-23T01:19:01.7761466Z  distributed/fsdp/test_fsdp_uneven
2022-11-23T01:19:01.7761863Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-11-23T01:19:01.7762273Z  distributed/_shard/sharded_tensor/ops/test_embedding
2022-11-23T01:19:01.7762703Z  distributed/_shard/sharded_tensor/ops/test_chunk
2022-11-23T01:19:01.7763098Z  distributed/test_c10d_error_logger
2022-11-23T01:19:01.7763488Z  distributed/_shard/sharding_spec/test_sharding_spec
2022-11-23T01:19:01.7763925Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T01:19:01.7764360Z  distributed/_tensor/parallel/test_parallelize_api
2022-11-23T01:19:01.7764749Z  distributed/_tensor/test_math_ops
2022-11-23T01:19:01.7765156Z  distributed/_shard/sharded_tensor/ops/test_tensor_ops
2022-11-23T01:19:01.7765593Z  distributed/_shard/sharding_plan/test_sharding_plan
2022-11-23T01:19:01.7766287Z  distributed/fsdp/test_fsdp_memory
2022-11-23T01:19:01.7766690Z  distributed/fsdp/test_fsdp_tp_integration
2022-11-23T01:19:01.7767091Z  distributed/_composable/test_fully_shard
2022-11-23T01:19:01.7767455Z  distributed/test_c10d_spawn_ucc
2022-11-23T01:19:01.7767874Z  distributed/fsdp/test_fsdp_clip_grad_norm
2022-11-23T01:19:01.7768291Z  distributed/elastic/multiprocessing/api_test
2022-11-23T01:19:01.7768694Z  distributed/_tensor/test_matrix_ops
2022-11-23T01:19:01.7769061Z  distributed/fsdp/test_fsdp_exec_order
2022-11-23T01:19:01.7769438Z  distributed/test_c10d_spawn_gloo
2022-11-23T01:19:01.7769801Z  distributed/test_c10d_spawn_nccl
2022-11-23T01:19:01.7770156Z  distributed/fsdp/test_fsdp_grad_acc
2022-11-23T01:19:01.7770611Z  distributed/test_store
2022-11-23T01:19:01.7770970Z  distributed/fsdp/test_wrap
2022-11-23T01:19:01.7771351Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T01:19:01.7771773Z  distributed/fsdp/test_fsdp_use_orig_params
2022-11-23T01:19:01.7772178Z  distributed/fsdp/test_fsdp_mixed_precision
2022-11-23T01:19:01.7772571Z  distributed/rpc/cuda/test_tensorpipe_agent
2022-11-23T01:19:04.0368480Z Ignoring disabled issues:  []
2022-11-23T01:19:04.0403955Z Ignoring disabled issues:  []
2022-11-23T01:19:04.4474489Z Running distributed/algorithms/quantization/test_quantization ... [2022-11-23 01:19:04.446966]
2022-11-23T01:19:04.4482502Z /usr/bin/mpiexec
2022-11-23T01:19:04.4482982Z MPI not available -- MPI backend tests will be skipped
2022-11-23T01:19:04.4483645Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2}
2022-11-23T01:19:04.4488626Z Running distributed tests for the test backend with env init_method in shard 1 of 3
2022-11-23T01:19:04.4493651Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:19:04.449065]
2022-11-23T01:19:06.5951395Z 
2022-11-23T01:19:06.5951989Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:19:06.5958229Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization__oqz13ws)
2022-11-23T01:19:06.5958879Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:19:06.5959123Z 
2022-11-23T01:19:06.5959500Z ##[endgroup]
2022-11-23T01:19:06.5960414Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization__oqz13ws)
2022-11-23T01:19:06.5960910Z 
2022-11-23T01:19:06.5962740Z Running distributed tests for the test backend with file init_method in shard 1 of 3
2022-11-23T01:19:06.5968055Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:19:06.596414]
2022-11-23T01:19:08.7097602Z 
2022-11-23T01:19:08.7098118Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:19:08.7099302Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_u6pmu0tm)
2022-11-23T01:19:08.7099946Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:19:08.7100193Z 
2022-11-23T01:19:08.7100552Z ##[endgroup]
2022-11-23T01:19:08.7101477Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_u6pmu0tm)
2022-11-23T01:19:08.7101963Z 
2022-11-23T01:19:08.7106643Z Shard 1: nccl should be run in 2
2022-11-23T01:19:08.7107204Z Running distributed tests for the gloo backend with env init_method in shard 1 of 3
2022-11-23T01:19:08.7112475Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:19:08.710934]
2022-11-23T01:19:31.6204483Z 
2022-11-23T01:19:31.6208089Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:19:31.6209917Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_kwstgyge)
2022-11-23T01:19:31.6212128Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-11-23T01:19:31.6213688Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:31.6218561Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:31.6219040Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:31.6219498Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:31.6219968Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:31.6220417Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:31.6221082Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:19:31.6221858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:31.6222413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:31.6223071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:31.6223638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:31.6223923Z 
2022-11-23T01:19:31.6226493Z Running tests...
2022-11-23T01:19:31.6227066Z ----------------------------------------------------------------------
2022-11-23T01:19:31.6227773Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:31.6228421Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:19:31.6228966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 977
2022-11-23T01:19:31.6229506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 978
2022-11-23T01:19:31.6230208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:31.6230747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:31.6231385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:31.6232154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:31.6232824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:31.6233360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:31.6234025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:31.6234582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:31.6235105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:31.6235667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:31.6236212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:31.6236795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:31.6237535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:31.6238321Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:31.6238785Z ok (3.933s)
2022-11-23T01:19:31.6238969Z 
2022-11-23T01:19:31.6239271Z ----------------------------------------------------------------------
2022-11-23T01:19:31.6239682Z Ran 1 test in 3.933s
2022-11-23T01:19:31.6239881Z 
2022-11-23T01:19:31.6240066Z OK
2022-11-23T01:19:31.6240250Z 
2022-11-23T01:19:31.6240410Z Generating XML reports...
2022-11-23T01:19:31.6241152Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011912.xml
2022-11-23T01:19:31.6241999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:31.6242524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:31.6243181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:31.6243728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:31.6243998Z 
2022-11-23T01:19:31.6244127Z Running tests...
2022-11-23T01:19:31.6244611Z ----------------------------------------------------------------------
2022-11-23T01:19:31.6245294Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:31.6246674Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:19:31.6247216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1086
2022-11-23T01:19:31.6247744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1087
2022-11-23T01:19:31.6248452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:31.6248960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:31.6249614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:31.6250163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:31.6250822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:31.6251330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:31.6251982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:31.6252521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:31.6253164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:31.6253697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:31.6254256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:31.6254826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:31.6255623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:31.6256386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:31.6256862Z ok (3.851s)
2022-11-23T01:19:31.6257044Z 
2022-11-23T01:19:31.6257348Z ----------------------------------------------------------------------
2022-11-23T01:19:31.6257753Z Ran 1 test in 3.852s
2022-11-23T01:19:31.6257929Z 
2022-11-23T01:19:31.6258064Z OK
2022-11-23T01:19:31.6258230Z 
2022-11-23T01:19:31.6258385Z Generating XML reports...
2022-11-23T01:19:31.6259123Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011918.xml
2022-11-23T01:19:31.6260205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:31.6260733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:31.6261487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:31.6262051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:31.6262319Z 
2022-11-23T01:19:31.6262447Z Running tests...
2022-11-23T01:19:31.6262922Z ----------------------------------------------------------------------
2022-11-23T01:19:31.6263604Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:31.6264246Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-11-23T01:19:31.6264555Z 
2022-11-23T01:19:31.6264854Z ----------------------------------------------------------------------
2022-11-23T01:19:31.6265255Z Ran 1 test in 0.001s
2022-11-23T01:19:31.6265448Z 
2022-11-23T01:19:31.6265638Z OK (skipped=1)
2022-11-23T01:19:31.6265829Z 
2022-11-23T01:19:31.6265963Z Generating XML reports...
2022-11-23T01:19:31.6266698Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011924.xml
2022-11-23T01:19:31.6267538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:31.6268056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:31.6268695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:31.6269241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:31.6269508Z 
2022-11-23T01:19:31.6269654Z Running tests...
2022-11-23T01:19:31.6270105Z ----------------------------------------------------------------------
2022-11-23T01:19:31.6270781Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:31.6271422Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-11-23T01:19:31.6271751Z 
2022-11-23T01:19:31.6272049Z ----------------------------------------------------------------------
2022-11-23T01:19:31.6272419Z Ran 1 test in 0.001s
2022-11-23T01:19:31.6272614Z 
2022-11-23T01:19:31.6272760Z OK (skipped=1)
2022-11-23T01:19:31.6272945Z 
2022-11-23T01:19:31.6273171Z Generating XML reports...
2022-11-23T01:19:31.6273886Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011926.xml
2022-11-23T01:19:31.6274721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:31.6275247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:31.6275888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:31.6276414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:31.6276688Z 
2022-11-23T01:19:31.6276838Z Running tests...
2022-11-23T01:19:31.6277306Z ----------------------------------------------------------------------
2022-11-23T01:19:31.6277985Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:31.6278633Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s)
2022-11-23T01:19:31.6278984Z 
2022-11-23T01:19:31.6279283Z ----------------------------------------------------------------------
2022-11-23T01:19:31.6279678Z Ran 1 test in 0.001s
2022-11-23T01:19:31.6279875Z 
2022-11-23T01:19:31.6280001Z OK (skipped=1)
2022-11-23T01:19:31.6280193Z 
2022-11-23T01:19:31.6280352Z Generating XML reports...
2022-11-23T01:19:31.6281080Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011928.xml
2022-11-23T01:19:31.6281982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:31.6282499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:31.6283154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:31.6283706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:31.6283975Z 
2022-11-23T01:19:31.6284121Z Running tests...
2022-11-23T01:19:31.6284570Z ----------------------------------------------------------------------
2022-11-23T01:19:31.6285242Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:31.6286587Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s)
2022-11-23T01:19:31.6286950Z 
2022-11-23T01:19:31.6287241Z ----------------------------------------------------------------------
2022-11-23T01:19:31.6287634Z Ran 1 test in 0.001s
2022-11-23T01:19:31.6287830Z 
2022-11-23T01:19:31.6287975Z OK (skipped=1)
2022-11-23T01:19:31.6288163Z 
2022-11-23T01:19:31.6288317Z Generating XML reports...
2022-11-23T01:19:31.6289031Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011931.xml
2022-11-23T01:19:31.6289488Z 
2022-11-23T01:19:31.6290060Z ##[endgroup]
2022-11-23T01:19:31.6290846Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_kwstgyge)
2022-11-23T01:19:31.6291336Z 
2022-11-23T01:19:31.6291585Z Running distributed tests for the gloo backend with file init_method in shard 1 of 3
2022-11-23T01:19:31.6292442Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:19:31.621316]
2022-11-23T01:19:54.6306079Z 
2022-11-23T01:19:54.6308621Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:19:54.6310183Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_2k62vri1)
2022-11-23T01:19:54.6312457Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-11-23T01:19:54.6313571Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:54.6314021Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:54.6314462Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:54.6314888Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:54.6315313Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:54.6315775Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:19:54.6316366Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:19:54.6317136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:54.6317650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:54.6318319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:54.6318863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:54.6319145Z 
2022-11-23T01:19:54.6319422Z Running tests...
2022-11-23T01:19:54.6319910Z ----------------------------------------------------------------------
2022-11-23T01:19:54.6320593Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:54.6321947Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:19:54.6322496Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1360
2022-11-23T01:19:54.6323014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1361
2022-11-23T01:19:54.6323734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:54.6324264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:54.6324901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:54.6325455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:54.6326772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:54.6327300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:54.6327935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:54.6328474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:54.6328980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:54.6329511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:54.6330072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:54.6330644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:54.6331381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:54.6332137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:54.6332769Z ok (3.927s)
2022-11-23T01:19:54.6332951Z 
2022-11-23T01:19:54.6333254Z ----------------------------------------------------------------------
2022-11-23T01:19:54.6333649Z Ran 1 test in 3.928s
2022-11-23T01:19:54.6333823Z 
2022-11-23T01:19:54.6333945Z OK
2022-11-23T01:19:54.6334112Z 
2022-11-23T01:19:54.6334264Z Generating XML reports...
2022-11-23T01:19:54.6334995Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011935.xml
2022-11-23T01:19:54.6335821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:54.6336344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:54.6336993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:54.6337534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:54.6337788Z 
2022-11-23T01:19:54.6337933Z Running tests...
2022-11-23T01:19:54.6338396Z ----------------------------------------------------------------------
2022-11-23T01:19:54.6339065Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:54.6339670Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:19:54.6340212Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1469
2022-11-23T01:19:54.6340778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1470
2022-11-23T01:19:54.6341540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:54.6342078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:54.6342707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:54.6343251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:54.6343904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:54.6344416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:54.6345042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:54.6345581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:54.6346091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:19:54.6346623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:19:54.6347176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:19:54.6347744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:19:54.6348481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:54.6349233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:19:54.6349705Z ok (3.944s)
2022-11-23T01:19:54.6349885Z 
2022-11-23T01:19:54.6350181Z ----------------------------------------------------------------------
2022-11-23T01:19:54.6350556Z Ran 1 test in 3.944s
2022-11-23T01:19:54.6350752Z 
2022-11-23T01:19:54.6350882Z OK
2022-11-23T01:19:54.6351049Z 
2022-11-23T01:19:54.6351202Z Generating XML reports...
2022-11-23T01:19:54.6351930Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011941.xml
2022-11-23T01:19:54.6352751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:54.6353344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:54.6353990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:54.6354538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:54.6354791Z 
2022-11-23T01:19:54.6354934Z Running tests...
2022-11-23T01:19:54.6355398Z ----------------------------------------------------------------------
2022-11-23T01:19:54.6356072Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:54.6356696Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-11-23T01:19:54.6357027Z 
2022-11-23T01:19:54.6357316Z ----------------------------------------------------------------------
2022-11-23T01:19:54.6357711Z Ran 1 test in 0.001s
2022-11-23T01:19:54.6357911Z 
2022-11-23T01:19:54.6358056Z OK (skipped=1)
2022-11-23T01:19:54.6358222Z 
2022-11-23T01:19:54.6358375Z Generating XML reports...
2022-11-23T01:19:54.6359099Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011947.xml
2022-11-23T01:19:54.6359931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:54.6360440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:54.6361157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:54.6361715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:54.6361981Z 
2022-11-23T01:19:54.6362125Z Running tests...
2022-11-23T01:19:54.6362575Z ----------------------------------------------------------------------
2022-11-23T01:19:54.6363251Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:54.6363935Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-11-23T01:19:54.6364266Z 
2022-11-23T01:19:54.6364563Z ----------------------------------------------------------------------
2022-11-23T01:19:54.6364927Z Ran 1 test in 0.001s
2022-11-23T01:19:54.6365120Z 
2022-11-23T01:19:54.6365264Z OK (skipped=1)
2022-11-23T01:19:54.6365448Z 
2022-11-23T01:19:54.6365601Z Generating XML reports...
2022-11-23T01:19:54.6366952Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011949.xml
2022-11-23T01:19:54.6367789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:54.6368312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:54.6368960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:54.6369486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:54.6369754Z 
2022-11-23T01:19:54.6369900Z Running tests...
2022-11-23T01:19:54.6370364Z ----------------------------------------------------------------------
2022-11-23T01:19:54.6371020Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:54.6371683Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s)
2022-11-23T01:19:54.6372034Z 
2022-11-23T01:19:54.6372327Z ----------------------------------------------------------------------
2022-11-23T01:19:54.6372716Z Ran 1 test in 0.001s
2022-11-23T01:19:54.6372892Z 
2022-11-23T01:19:54.6373036Z OK (skipped=1)
2022-11-23T01:19:54.6373221Z 
2022-11-23T01:19:54.6373472Z Generating XML reports...
2022-11-23T01:19:54.6374202Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011951.xml
2022-11-23T01:19:54.6375033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:19:54.6375539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:19:54.6376186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:19:54.6376731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:19:54.6377002Z 
2022-11-23T01:19:54.6377130Z Running tests...
2022-11-23T01:19:54.6377591Z ----------------------------------------------------------------------
2022-11-23T01:19:54.6378254Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:19:54.6378912Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s)
2022-11-23T01:19:54.6379238Z 
2022-11-23T01:19:54.6379532Z ----------------------------------------------------------------------
2022-11-23T01:19:54.6379920Z Ran 1 test in 0.001s
2022-11-23T01:19:54.6380112Z 
2022-11-23T01:19:54.6380258Z OK (skipped=1)
2022-11-23T01:19:54.6380441Z 
2022-11-23T01:19:54.6380577Z Generating XML reports...
2022-11-23T01:19:54.6381296Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011954.xml
2022-11-23T01:19:54.6381818Z 
2022-11-23T01:19:54.6382233Z ##[endgroup]
2022-11-23T01:19:54.6383001Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_2k62vri1)
2022-11-23T01:19:54.6383490Z 
2022-11-23T01:19:54.6383740Z Running distributed tests for the ucc backend with env init_method in shard 1 of 3
2022-11-23T01:19:54.6384593Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:19:54.631788]
2022-11-23T01:19:56.7443936Z 
2022-11-23T01:19:56.7444843Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:19:56.7446802Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_lt1dimyr)
2022-11-23T01:19:56.7447462Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:19:56.7447704Z 
2022-11-23T01:19:56.7448101Z ##[endgroup]
2022-11-23T01:19:56.7449000Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_lt1dimyr)
2022-11-23T01:19:56.7449485Z 
2022-11-23T01:19:56.7455393Z Running distributed tests for the ucc backend with file init_method in shard 1 of 3
2022-11-23T01:19:56.7459811Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:19:56.745657]
2022-11-23T01:19:58.8488179Z 
2022-11-23T01:19:58.8489670Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:19:58.8491898Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_xyqp5cr7)
2022-11-23T01:19:58.8493114Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:19:58.8493542Z 
2022-11-23T01:19:58.8494204Z ##[endgroup]
2022-11-23T01:19:58.8495828Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_xyqp5cr7)
2022-11-23T01:19:58.8497081Z 
2022-11-23T01:19:58.8501578Z Running distributed/test_distributed_spawn ... [2022-11-23 01:19:58.849771]
2022-11-23T01:19:58.8509839Z /usr/bin/mpiexec
2022-11-23T01:19:58.8510676Z MPI not available -- MPI backend tests will be skipped
2022-11-23T01:19:58.8511887Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 3}
2022-11-23T01:19:58.8512947Z Running distributed tests for the test backend with env init_method in shard 1 of 3
2022-11-23T01:19:58.8516938Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:19:58.851311]
2022-11-23T01:20:01.2497483Z 
2022-11-23T01:20:01.2498210Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T01:20:01.2499278Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_zvbs2fen)
2022-11-23T01:20:01.2499829Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:20:01.2500074Z 
2022-11-23T01:20:01.2500464Z ##[endgroup]
2022-11-23T01:20:01.2501247Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_zvbs2fen)
2022-11-23T01:20:01.2501656Z 
2022-11-23T01:20:01.2507472Z Running distributed tests for the test backend with file init_method in shard 1 of 3
2022-11-23T01:20:01.2511929Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:20:01.250870]
2022-11-23T01:20:03.6688460Z 
2022-11-23T01:20:03.6689247Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T01:20:03.6692402Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_j4gzt5fx)
2022-11-23T01:20:03.6693111Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:20:03.6693374Z 
2022-11-23T01:20:03.6693759Z ##[endgroup]
2022-11-23T01:20:03.6694541Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_j4gzt5fx)
2022-11-23T01:20:03.6694949Z 
2022-11-23T01:20:03.6697579Z Shard 1: nccl should be run in 2
2022-11-23T01:20:03.6698428Z Running distributed tests for the gloo backend with env init_method in shard 1 of 3
2022-11-23T01:20:03.6703720Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:20:03.670011]
2022-11-23T01:48:20.9182038Z 
2022-11-23T01:48:20.9182557Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T01:48:20.9183640Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_gcmccc03)
2022-11-23T01:48:20.9191783Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:48:20.9243821Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-11-23T01:48:20.9290439Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9291278Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9291979Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9297406Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9298372Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9299143Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9299699Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9300480Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9301413Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9302513Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9303187Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9304200Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9305203Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9306089Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9306641Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9307210Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9307917Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9308623Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9309423Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9310094Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9311016Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9311789Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9312292Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9313006Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9313915Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9314711Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9315368Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9316226Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9316989Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9317586Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9318052Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9318505Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9318994Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9319586Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9320101Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9320573Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9321055Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9321554Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9322030Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9322519Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9323011Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9323532Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9324038Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9324760Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9325332Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9326359Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9326907Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9327434Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9327929Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9328429Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9328951Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9329437Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9329919Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9330419Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9330925Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9331386Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9331883Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9332471Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9332953Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9333396Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9333868Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9334348Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9334781Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9335233Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9335703Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9336181Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9336642Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9337114Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9337582Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9338021Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9338496Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9338966Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9339420Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9339911Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9340426Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9340950Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9341390Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9341849Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9342329Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9342780Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9343255Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9343727Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9344191Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9344699Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9345219Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9345750Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9346260Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9346808Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9347328Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9347853Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9348349Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9348868Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9349388Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9349895Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9350445Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9350989Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9351525Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9352043Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9352550Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9353078Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9353521Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9353960Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9354407Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9354840Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9355319Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9355777Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9356238Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9356692Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9357193Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9357659Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9358128Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9358607Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9359111Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9359610Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9360095Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9360581Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9361071Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9361609Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9362120Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9362610Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9363074Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9363522Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9363973Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9364442Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9364892Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9365358Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9366303Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9367094Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9367620Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9368185Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9368690Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9369193Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9369717Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9370269Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9370775Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9371272Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9371790Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9372284Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9372722Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9373180Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9373879Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9374738Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9375599Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9376533Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9377436Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9378353Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9379363Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9380481Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9381621Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9382886Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9384190Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9385464Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9386685Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9388157Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9389420Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9390691Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9391866Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9392906Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9393865Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9394719Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9395562Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9396444Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9397325Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9398237Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9399193Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9400203Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9401257Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9402143Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9402992Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9403904Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9404830Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9406147Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9407052Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9407953Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9408865Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9409801Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9410927Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9411790Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9412694Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9413618Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9414459Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9415377Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9416382Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9417325Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9418178Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9419014Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9419886Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9420791Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9421671Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9422504Z test_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9423310Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9424089Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9424910Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9425914Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9426709Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9427545Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9428383Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9429165Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9429942Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9430778Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9431625Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9432484Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9433308Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9434065Z test_isend (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9434861Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9435743Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9436660Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9437622Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9438612Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9439528Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9440447Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9441406Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9442337Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9443236Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9444097Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9445001Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9446242Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9447113Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9447937Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9448745Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9449937Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9450950Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9451962Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9452934Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9453892Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9454884Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9455837Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9456762Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9457662Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9458612Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9459561Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9460576Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9461682Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9462741Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9463813Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9464651Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9465523Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9466396Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9467213Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9468036Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9468890Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9469705Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9470521Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9471320Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9472125Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9472923Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9473797Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9474672Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9475477Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9476278Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9477141Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9478057Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9478849Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9479693Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9480523Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9481319Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9482159Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9483007Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9483826Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9484666Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9485471Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9486626Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9487736Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9488719Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9489642Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9490474Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9491353Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9492287Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9493173Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9494029Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9494908Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9495827Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9496747Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9497642Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9498538Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9499383Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9500225Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9501132Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9502082Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9503234Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:20.9504714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9505680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9506900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9507916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9508416Z 
2022-11-23T01:48:20.9508664Z Running tests...
2022-11-23T01:48:20.9509511Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9510643Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9511872Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:20.9513090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1876
2022-11-23T01:48:20.9514029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1877
2022-11-23T01:48:20.9515305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9516287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9517514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9518531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9519799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9520768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9521997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9523028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9523986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:20.9525197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:20.9526958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9528447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9529561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:20.9530546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:20.9531719Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:20.9533469Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:20.9534879Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:20.9536643Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:20.9538078Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:20.9539816Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:20.9541457Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:20.9543250Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:20.9544654Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:20.9546380Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:20.9547765Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:20.9549539Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:20.9550900Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:20.9552533Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:20.9553939Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:48:20.9555722Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:48:20.9556763Z ok (5.667s)
2022-11-23T01:48:20.9557092Z 
2022-11-23T01:48:20.9557651Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9558352Z Ran 1 test in 5.667s
2022-11-23T01:48:20.9558681Z 
2022-11-23T01:48:20.9558875Z OK
2022-11-23T01:48:20.9559170Z 
2022-11-23T01:48:20.9559445Z Generating XML reports...
2022-11-23T01:48:20.9560742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012007.xml
2022-11-23T01:48:20.9562263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9563435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9564723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9566023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9566520Z 
2022-11-23T01:48:20.9566770Z Running tests...
2022-11-23T01:48:20.9567607Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9568743Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9569886Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-11-23T01:48:20.9570540Z 
2022-11-23T01:48:20.9571076Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9571794Z Ran 1 test in 0.004s
2022-11-23T01:48:20.9572140Z 
2022-11-23T01:48:20.9572388Z OK (skipped=1)
2022-11-23T01:48:20.9572730Z 
2022-11-23T01:48:20.9573007Z Generating XML reports...
2022-11-23T01:48:20.9574269Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012016.xml
2022-11-23T01:48:20.9575799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9576765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9577968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9579151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9579644Z 
2022-11-23T01:48:20.9579895Z Running tests...
2022-11-23T01:48:20.9580773Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9581878Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9582976Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:20.9584032Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2021
2022-11-23T01:48:20.9584965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2022
2022-11-23T01:48:20.9586272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9587210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9588460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9589447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9590697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9591680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9592912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9593922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9594878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:20.9596013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:20.9597394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9598868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9599988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:20.9601017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:20.9601953Z ok (4.155s)
2022-11-23T01:48:20.9602285Z 
2022-11-23T01:48:20.9602865Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9603564Z Ran 1 test in 4.155s
2022-11-23T01:48:20.9603909Z 
2022-11-23T01:48:20.9604096Z OK
2022-11-23T01:48:20.9604395Z 
2022-11-23T01:48:20.9604680Z Generating XML reports...
2022-11-23T01:48:20.9606286Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012018.xml
2022-11-23T01:48:20.9607801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9608776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9610016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9611000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9611511Z 
2022-11-23T01:48:20.9611755Z Running tests...
2022-11-23T01:48:20.9612614Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9613760Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9614871Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:20.9617212Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.620s)
2022-11-23T01:48:20.9618344Z 
2022-11-23T01:48:20.9618892Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9619615Z Ran 1 test in 1.620s
2022-11-23T01:48:20.9619987Z 
2022-11-23T01:48:20.9620229Z OK (skipped=1)
2022-11-23T01:48:20.9620567Z 
2022-11-23T01:48:20.9620815Z Generating XML reports...
2022-11-23T01:48:20.9622107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012025.xml
2022-11-23T01:48:20.9623624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9624579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9625817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9626857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9627357Z 
2022-11-23T01:48:20.9627607Z Running tests...
2022-11-23T01:48:20.9628492Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9629592Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9630752Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:20.9631859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2164
2022-11-23T01:48:20.9632801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2165
2022-11-23T01:48:20.9634104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9635094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9636354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9637360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9638638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9639799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9641055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9642051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9643016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:20.9644108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:20.9645502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9647218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9648373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:20.9649415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:20.9650453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwtsvc41
2022-11-23T01:48:20.9651592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7fok0bu
2022-11-23T01:48:20.9652727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwtsvc41/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9653894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7fok0bu/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9655177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9656259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9657036Z ok (4.243s)
2022-11-23T01:48:20.9657344Z 
2022-11-23T01:48:20.9657940Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9658681Z Ran 1 test in 4.243s
2022-11-23T01:48:20.9659031Z 
2022-11-23T01:48:20.9659248Z OK
2022-11-23T01:48:20.9659547Z 
2022-11-23T01:48:20.9659799Z Generating XML reports...
2022-11-23T01:48:20.9661142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012029.xml
2022-11-23T01:48:20.9662682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9663645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9664869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9665903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9666391Z 
2022-11-23T01:48:20.9666634Z Running tests...
2022-11-23T01:48:20.9667483Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9668635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9669832Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:20.9670977Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2277
2022-11-23T01:48:20.9671917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2278
2022-11-23T01:48:20.9673222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9674215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9675455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9676461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9677893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9678870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9680086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9681098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9682055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:20.9683121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:20.9684512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9686368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9687501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:20.9688587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppco8d5xx
2022-11-23T01:48:20.9689752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppco8d5xx/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9690867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:20.9691955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpe3_spj
2022-11-23T01:48:20.9693256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpe3_spj/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9694378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9695411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9696201Z ok (4.360s)
2022-11-23T01:48:20.9696499Z 
2022-11-23T01:48:20.9697084Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9697807Z Ran 1 test in 4.361s
2022-11-23T01:48:20.9698159Z 
2022-11-23T01:48:20.9698377Z OK
2022-11-23T01:48:20.9698664Z 
2022-11-23T01:48:20.9698925Z Generating XML reports...
2022-11-23T01:48:20.9700220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012036.xml
2022-11-23T01:48:20.9701743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9702734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9703983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9704986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9705485Z 
2022-11-23T01:48:20.9705743Z Running tests...
2022-11-23T01:48:20.9706583Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9707711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9708880Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:20.9709988Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2390
2022-11-23T01:48:20.9710941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2391
2022-11-23T01:48:20.9712254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9713232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9714457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9715670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9716954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9717913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9719163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9720181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9721137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:20.9722150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:20.9723568Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9725043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9726530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:20.9727518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:20.9728565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpclcjls2b
2022-11-23T01:48:20.9729676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvfqlwqb
2022-11-23T01:48:20.9730998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpclcjls2b/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9732151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvfqlwqb/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9733270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9734294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9735324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9736363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9737406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9738447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9739467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9740598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9741638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9742676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9743720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9744771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9745554Z ok (6.056s)
2022-11-23T01:48:20.9745881Z 
2022-11-23T01:48:20.9746443Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9747143Z Ran 1 test in 6.056s
2022-11-23T01:48:20.9747486Z 
2022-11-23T01:48:20.9747696Z OK
2022-11-23T01:48:20.9747995Z 
2022-11-23T01:48:20.9748243Z Generating XML reports...
2022-11-23T01:48:20.9749559Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012043.xml
2022-11-23T01:48:20.9751093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9752045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9753274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9754483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9754985Z 
2022-11-23T01:48:20.9755235Z Running tests...
2022-11-23T01:48:20.9756118Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9757209Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9758371Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:20.9759531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2505
2022-11-23T01:48:20.9760496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2506
2022-11-23T01:48:20.9761804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9762793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9764031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9765038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9766616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9767585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9768972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9769999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9770985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:20.9772054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:20.9773443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9774929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9776057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:20.9777070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:20.9778122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvwsm54ef
2022-11-23T01:48:20.9779271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvwsm54ef/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9780422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxtv6q5d
2022-11-23T01:48:20.9781539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxtv6q5d/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9782651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9783700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9784736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9785764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9786547Z ok (5.046s)
2022-11-23T01:48:20.9786876Z 
2022-11-23T01:48:20.9787464Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9788138Z Ran 1 test in 5.046s
2022-11-23T01:48:20.9788479Z 
2022-11-23T01:48:20.9788688Z OK
2022-11-23T01:48:20.9788996Z 
2022-11-23T01:48:20.9789271Z Generating XML reports...
2022-11-23T01:48:20.9790572Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012051.xml
2022-11-23T01:48:20.9792289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9793251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9794490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9795505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9795982Z 
2022-11-23T01:48:20.9796234Z Running tests...
2022-11-23T01:48:20.9797132Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9798255Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9799463Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:20.9800661Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2620
2022-11-23T01:48:20.9801650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2621
2022-11-23T01:48:20.9802974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9803931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9805192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9806486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9807882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9808873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9810117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9811149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9812106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:20.9813186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:20.9814634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9816132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9817257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:20.9818287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:20.9819378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3ccx8fk
2022-11-23T01:48:20.9820533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3ccx8fk/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9821694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnldo84cg
2022-11-23T01:48:20.9822854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnldo84cg/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9823979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9825018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9826083Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9827129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9828151Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9829394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9830441Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9831489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9832242Z ok (5.250s)
2022-11-23T01:48:20.9832570Z 
2022-11-23T01:48:20.9833142Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9833861Z Ran 1 test in 5.251s
2022-11-23T01:48:20.9834211Z 
2022-11-23T01:48:20.9834399Z OK
2022-11-23T01:48:20.9834694Z 
2022-11-23T01:48:20.9834987Z Generating XML reports...
2022-11-23T01:48:20.9836297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012059.xml
2022-11-23T01:48:20.9837806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9838780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9840030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9841111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9841624Z 
2022-11-23T01:48:20.9841871Z Running tests...
2022-11-23T01:48:20.9842722Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9843858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9845265Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:20.9846741Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2735
2022-11-23T01:48:20.9847731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2736
2022-11-23T01:48:20.9849075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9850060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9851285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9852304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9853585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9854527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9855785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9856815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9857791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:20.9858843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:20.9860256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9861721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9862844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:20.9863834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:20.9864921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3o47fkd
2022-11-23T01:48:20.9866074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3o47fkd/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9867361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbxmtzsg
2022-11-23T01:48:20.9868559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbxmtzsg/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9869659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9870695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9871453Z ok (5.353s)
2022-11-23T01:48:20.9871776Z 
2022-11-23T01:48:20.9872355Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9873072Z Ran 1 test in 5.353s
2022-11-23T01:48:20.9873422Z 
2022-11-23T01:48:20.9873604Z OK
2022-11-23T01:48:20.9873900Z 
2022-11-23T01:48:20.9874181Z Generating XML reports...
2022-11-23T01:48:20.9875483Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012106.xml
2022-11-23T01:48:20.9877028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9877961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9879198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9880214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9880718Z 
2022-11-23T01:48:20.9880941Z Running tests...
2022-11-23T01:48:20.9881801Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9883106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9884382Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:20.9885566Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2850
2022-11-23T01:48:20.9886875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2851
2022-11-23T01:48:20.9888185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9889152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9890374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9891376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9892635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9893574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9894813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9895916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9896889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:20.9897886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:20.9899237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9900700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9901840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:20.9902850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:20.9903936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpxgwmdh
2022-11-23T01:48:20.9905300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpxgwmdh/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9906383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb120d8bt
2022-11-23T01:48:20.9907531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb120d8bt/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9908628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9909679Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9910687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9911695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9912465Z ok (5.757s)
2022-11-23T01:48:20.9912795Z 
2022-11-23T01:48:20.9913364Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9914086Z Ran 1 test in 5.757s
2022-11-23T01:48:20.9914430Z 
2022-11-23T01:48:20.9914639Z OK
2022-11-23T01:48:20.9914933Z 
2022-11-23T01:48:20.9915210Z Generating XML reports...
2022-11-23T01:48:20.9916459Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012114.xml
2022-11-23T01:48:20.9917992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9918965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9920330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9921353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9921839Z 
2022-11-23T01:48:20.9922082Z Running tests...
2022-11-23T01:48:20.9922940Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9924046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9925235Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:20.9926658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2965
2022-11-23T01:48:20.9927600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2966
2022-11-23T01:48:20.9928918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9929886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9931122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9932100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9933330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9934303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9935550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9936558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9937539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:20.9938590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:20.9940028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9941558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9942889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:20.9943909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:20.9944940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9l9n0lbh
2022-11-23T01:48:20.9946090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9l9n0lbh/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9947221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpag27bh48
2022-11-23T01:48:20.9948363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpag27bh48/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9949417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9950482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9951518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9952534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9953317Z ok (5.549s)
2022-11-23T01:48:20.9953631Z 
2022-11-23T01:48:20.9954231Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9954946Z Ran 1 test in 5.550s
2022-11-23T01:48:20.9955287Z 
2022-11-23T01:48:20.9955500Z OK
2022-11-23T01:48:20.9955789Z 
2022-11-23T01:48:20.9956061Z Generating XML reports...
2022-11-23T01:48:20.9957352Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012123.xml
2022-11-23T01:48:20.9959065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9960060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9961314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9962343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9962811Z 
2022-11-23T01:48:20.9963064Z Running tests...
2022-11-23T01:48:20.9963953Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9965083Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:20.9966653Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:20.9967889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3080
2022-11-23T01:48:20.9968842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3081
2022-11-23T01:48:20.9970154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9971096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9972365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9973369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9974598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:20.9975577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:20.9976832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:20.9977858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:20.9978794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:20.9979866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:20.9981469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9982965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:20.9984080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:20.9985087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:20.9986160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuymx_kjq
2022-11-23T01:48:20.9987304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuymx_kjq/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9988431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8j_xecud
2022-11-23T01:48:20.9989580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8j_xecud/_remote_module_non_scriptable.py
2022-11-23T01:48:20.9990688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9991723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9992746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9993773Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:20.9994523Z ok (5.149s)
2022-11-23T01:48:20.9994841Z 
2022-11-23T01:48:20.9995423Z ----------------------------------------------------------------------
2022-11-23T01:48:20.9996296Z Ran 1 test in 5.149s
2022-11-23T01:48:20.9996661Z 
2022-11-23T01:48:20.9996870Z OK
2022-11-23T01:48:20.9997140Z 
2022-11-23T01:48:20.9997424Z Generating XML reports...
2022-11-23T01:48:20.9998727Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012131.xml
2022-11-23T01:48:21.0000250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0001204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0002468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0003492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0003977Z 
2022-11-23T01:48:21.0004223Z Running tests...
2022-11-23T01:48:21.0005071Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0006555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0007718Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0009950Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-11-23T01:48:21.0011041Z 
2022-11-23T01:48:21.0011591Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0012311Z Ran 1 test in 1.642s
2022-11-23T01:48:21.0012671Z 
2022-11-23T01:48:21.0012934Z OK (skipped=1)
2022-11-23T01:48:21.0013268Z 
2022-11-23T01:48:21.0013544Z Generating XML reports...
2022-11-23T01:48:21.0014813Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012138.xml
2022-11-23T01:48:21.0016326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0017301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0018728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0019724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0020205Z 
2022-11-23T01:48:21.0020450Z Running tests...
2022-11-23T01:48:21.0021330Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0022447Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0023609Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0024740Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3229
2022-11-23T01:48:21.0025681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3230
2022-11-23T01:48:21.0026989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0027963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0029197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0030159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0031516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0032487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0033882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0034881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0035830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0036880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0038401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0039911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0041113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0042143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0042895Z ok (4.365s)
2022-11-23T01:48:21.0043238Z 
2022-11-23T01:48:21.0043805Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0044517Z Ran 1 test in 4.365s
2022-11-23T01:48:21.0044860Z 
2022-11-23T01:48:21.0045051Z OK
2022-11-23T01:48:21.0045345Z 
2022-11-23T01:48:21.0045631Z Generating XML reports...
2022-11-23T01:48:21.0047318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012143.xml
2022-11-23T01:48:21.0048884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0049861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0051105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0052100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0052567Z 
2022-11-23T01:48:21.0052823Z Running tests...
2022-11-23T01:48:21.0053672Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0054812Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0056009Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0058437Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.589s)
2022-11-23T01:48:21.0059543Z 
2022-11-23T01:48:21.0060118Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0060826Z Ran 1 test in 1.590s
2022-11-23T01:48:21.0061165Z 
2022-11-23T01:48:21.0061405Z OK (skipped=1)
2022-11-23T01:48:21.0061743Z 
2022-11-23T01:48:21.0061998Z Generating XML reports...
2022-11-23T01:48:21.0063297Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012149.xml
2022-11-23T01:48:21.0064814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0065798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0067023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0068049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0068550Z 
2022-11-23T01:48:21.0068793Z Running tests...
2022-11-23T01:48:21.0069637Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0070778Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0072034Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0073133Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3372
2022-11-23T01:48:21.0074072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3373
2022-11-23T01:48:21.0075405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0076379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0077606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0078602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0079854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0080820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0082044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0083065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0084031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0085090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0086770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0088249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0089378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0090407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0091159Z ok (5.041s)
2022-11-23T01:48:21.0091478Z 
2022-11-23T01:48:21.0092054Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0092766Z Ran 1 test in 5.041s
2022-11-23T01:48:21.0093113Z 
2022-11-23T01:48:21.0093307Z OK
2022-11-23T01:48:21.0093791Z 
2022-11-23T01:48:21.0094063Z Generating XML reports...
2022-11-23T01:48:21.0095396Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012154.xml
2022-11-23T01:48:21.0096882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0097855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0099089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0100111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0100604Z 
2022-11-23T01:48:21.0100825Z Running tests...
2022-11-23T01:48:21.0101709Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0102868Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0104007Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-11-23T01:48:21.0104592Z 
2022-11-23T01:48:21.0105148Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0105825Z Ran 1 test in 0.002s
2022-11-23T01:48:21.0106175Z 
2022-11-23T01:48:21.0106417Z OK (skipped=1)
2022-11-23T01:48:21.0106750Z 
2022-11-23T01:48:21.0106998Z Generating XML reports...
2022-11-23T01:48:21.0108285Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012201.xml
2022-11-23T01:48:21.0109991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0110993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0112199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0113222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0113710Z 
2022-11-23T01:48:21.0113962Z Running tests...
2022-11-23T01:48:21.0114827Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0115969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0116979Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:21.0117992Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0119010Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3516
2022-11-23T01:48:21.0119970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3517
2022-11-23T01:48:21.0121300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0122242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0123513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0124529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0126125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0127057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0128305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0129303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0130233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0131231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0132835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0134307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0135411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0136416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0137483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_g7e1wy
2022-11-23T01:48:21.0138644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_g7e1wy/_remote_module_non_scriptable.py
2022-11-23T01:48:21.0139783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3pi3ynjw
2022-11-23T01:48:21.0141005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3pi3ynjw/_remote_module_non_scriptable.py
2022-11-23T01:48:21.0141864Z ok (4.269s)
2022-11-23T01:48:21.0142193Z 
2022-11-23T01:48:21.0142745Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0143455Z Ran 1 test in 4.269s
2022-11-23T01:48:21.0143813Z 
2022-11-23T01:48:21.0144025Z OK
2022-11-23T01:48:21.0144324Z 
2022-11-23T01:48:21.0144594Z Generating XML reports...
2022-11-23T01:48:21.0145870Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012204.xml
2022-11-23T01:48:21.0147649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0148645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0149881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0150885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0151381Z 
2022-11-23T01:48:21.0151637Z Running tests...
2022-11-23T01:48:21.0152567Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0153704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0154739Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:21.0155816Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0156884Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3629
2022-11-23T01:48:21.0157850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3630
2022-11-23T01:48:21.0159158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0160136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0161372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0162407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0163679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0164659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0166128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0167157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0168126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0169137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0170706Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0172161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0173262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0174232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0175285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw55cliy9
2022-11-23T01:48:21.0176419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw55cliy9/_remote_module_non_scriptable.py
2022-11-23T01:48:21.0177536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0hg7vod
2022-11-23T01:48:21.0178639Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0hg7vod/_remote_module_non_scriptable.py
2022-11-23T01:48:21.0179722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.0180736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.0181504Z ok (4.210s)
2022-11-23T01:48:21.0181822Z 
2022-11-23T01:48:21.0182412Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0183134Z Ran 1 test in 4.210s
2022-11-23T01:48:21.0183497Z 
2022-11-23T01:48:21.0183720Z OK
2022-11-23T01:48:21.0183994Z 
2022-11-23T01:48:21.0184284Z Generating XML reports...
2022-11-23T01:48:21.0185724Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012210.xml
2022-11-23T01:48:21.0187236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0188200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0189414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0190396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0190866Z 
2022-11-23T01:48:21.0191103Z Running tests...
2022-11-23T01:48:21.0191908Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0192920Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0193897Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:21.0194946Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0196076Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3742
2022-11-23T01:48:21.0196995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3743
2022-11-23T01:48:21.0198251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0199161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0200350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0201306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0202456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0203393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0204558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0205519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0206680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0207838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0209152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0210552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0211587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0212565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0213583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsn47c9cl
2022-11-23T01:48:21.0214648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6jn1xc7
2022-11-23T01:48:21.0215745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsn47c9cl/_remote_module_non_scriptable.py
2022-11-23T01:48:21.0216865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6jn1xc7/_remote_module_non_scriptable.py
2022-11-23T01:48:21.0217928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.0218914Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.0219653Z ok (4.249s)
2022-11-23T01:48:21.0219958Z 
2022-11-23T01:48:21.0220501Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0221173Z Ran 1 test in 4.249s
2022-11-23T01:48:21.0221596Z 
2022-11-23T01:48:21.0221809Z OK
2022-11-23T01:48:21.0222101Z 
2022-11-23T01:48:21.0222369Z Generating XML reports...
2022-11-23T01:48:21.0223659Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012217.xml
2022-11-23T01:48:21.0225128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0226094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0227310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0228303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0228761Z 
2022-11-23T01:48:21.0229001Z Running tests...
2022-11-23T01:48:21.0229844Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0230932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0231903Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:48:21.0232929Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0233944Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3855
2022-11-23T01:48:21.0234902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3856
2022-11-23T01:48:21.0236084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0236992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0238104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0238983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0240102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0241045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0242154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0243143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0244038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0245008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0246507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0247887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0248926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0249872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0250832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjk0_fmc
2022-11-23T01:48:21.0251881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppzteip2d
2022-11-23T01:48:21.0252951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjk0_fmc/_remote_module_non_scriptable.py
2022-11-23T01:48:21.0254040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppzteip2d/_remote_module_non_scriptable.py
2022-11-23T01:48:21.0254807Z ok (4.255s)
2022-11-23T01:48:21.0255109Z 
2022-11-23T01:48:21.0255631Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0256287Z Ran 1 test in 4.255s
2022-11-23T01:48:21.0256614Z 
2022-11-23T01:48:21.0256908Z OK
2022-11-23T01:48:21.0257196Z 
2022-11-23T01:48:21.0257457Z Generating XML reports...
2022-11-23T01:48:21.0258668Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012224.xml
2022-11-23T01:48:21.0260085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0260989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0262164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0263123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0263588Z 
2022-11-23T01:48:21.0263797Z Running tests...
2022-11-23T01:48:21.0264621Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0265693Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0266692Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0267649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3968
2022-11-23T01:48:21.0268565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3969
2022-11-23T01:48:21.0269811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0270723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0271895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0272860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0274051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0274955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0276150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0277120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0278194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0279189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0280531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0281950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0283010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0283998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0285583Z STAGE:2022-11-23 01:22:35 3968:3968 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:22:35 3969:3969 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0286571Z 
2022-11-23T01:48:21.0287687Z STAGE:2022-11-23 01:22:35 3968:3968 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:35 3969:3969 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0288473Z 
2022-11-23T01:48:21.0289633Z STAGE:2022-11-23 01:22:35 3968:3968 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:35 3969:3969 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0290444Z 
2022-11-23T01:48:21.0291095Z STAGE:2022-11-23 01:22:35 3968:3968 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0292393Z STAGE:2022-11-23 01:22:35 3969:3969 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0293555Z STAGE:2022-11-23 01:22:35 3969:3969 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0294710Z STAGE:2022-11-23 01:22:35 3968:3968 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0295890Z STAGE:2022-11-23 01:22:35 3969:3969 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0297096Z STAGE:2022-11-23 01:22:35 3968:3968 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0297833Z ok (4.248s)
2022-11-23T01:48:21.0298134Z 
2022-11-23T01:48:21.0298661Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0299311Z Ran 1 test in 4.248s
2022-11-23T01:48:21.0299641Z 
2022-11-23T01:48:21.0299840Z OK
2022-11-23T01:48:21.0300116Z 
2022-11-23T01:48:21.0300371Z Generating XML reports...
2022-11-23T01:48:21.0301582Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012231.xml
2022-11-23T01:48:21.0303018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0303940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0305085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0306058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0306522Z 
2022-11-23T01:48:21.0306755Z Running tests...
2022-11-23T01:48:21.0307583Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0308226Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0308764Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0309216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4081
2022-11-23T01:48:21.0309647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4082
2022-11-23T01:48:21.0310404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0310886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0311669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0312063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0312818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0313153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0313916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0314304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0314803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0315304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0316105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0316898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0317361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0318022Z STAGE:2022-11-23 01:22:41 4082:4082 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0319559Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0319811Z   warnings.warn(
2022-11-23T01:48:21.0320275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0320946Z STAGE:2022-11-23 01:22:42 4081:4081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0322422Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0322664Z   warnings.warn(
2022-11-23T01:48:21.0323763Z STAGE:2022-11-23 01:22:42 4082:4082 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:22:42 4081:4081 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0323796Z 
2022-11-23T01:48:21.0324921Z STAGE:2022-11-23 01:22:42 4082:4082 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:42 4081:4081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0324959Z 
2022-11-23T01:48:21.0325625Z STAGE:2022-11-23 01:22:42 4082:4082 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0326511Z STAGE:2022-11-23 01:22:42 4081:4081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0327179Z STAGE:2022-11-23 01:22:42 4082:4082 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0327816Z STAGE:2022-11-23 01:22:42 4081:4081 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0328526Z STAGE:2022-11-23 01:22:42 4082:4082 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0329218Z STAGE:2022-11-23 01:22:42 4081:4081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0329875Z STAGE:2022-11-23 01:22:42 4082:4082 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0330674Z STAGE:2022-11-23 01:22:42 4081:4081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0331347Z STAGE:2022-11-23 01:22:42 4082:4082 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0332021Z STAGE:2022-11-23 01:22:42 4081:4081 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0332714Z STAGE:2022-11-23 01:22:42 4082:4082 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0333407Z STAGE:2022-11-23 01:22:42 4081:4081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0333610Z ok (4.369s)
2022-11-23T01:48:21.0333642Z 
2022-11-23T01:48:21.0334177Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0334418Z Ran 1 test in 4.369s
2022-11-23T01:48:21.0334448Z 
2022-11-23T01:48:21.0334650Z OK
2022-11-23T01:48:21.0334677Z 
2022-11-23T01:48:21.0334940Z Generating XML reports...
2022-11-23T01:48:21.0335861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012238.xml
2022-11-23T01:48:21.0336636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0336997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0337785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0338160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0338188Z 
2022-11-23T01:48:21.0338536Z Running tests...
2022-11-23T01:48:21.0339080Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0339730Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0340299Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0340814Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4194
2022-11-23T01:48:21.0341264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4195
2022-11-23T01:48:21.0342025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0342363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0343149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0343558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0344321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0344685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0345475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0345872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0346367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0346863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0347663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0348490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0348977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0349459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0350055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.0350546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.0351373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0352197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0352884Z STAGE:2022-11-23 01:22:48 4194:4194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0353530Z STAGE:2022-11-23 01:22:48 4195:4195 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0355074Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0355335Z   warnings.warn(
2022-11-23T01:48:21.0356851Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0357103Z   warnings.warn(
2022-11-23T01:48:21.0357795Z STAGE:2022-11-23 01:22:48 4194:4194 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0358562Z STAGE:2022-11-23 01:22:48 4195:4195 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0359727Z STAGE:2022-11-23 01:22:48 4194:4194 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:22:48 4195:4195 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0359767Z 
2022-11-23T01:48:21.0360442Z STAGE:2022-11-23 01:22:48 4194:4194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0361108Z STAGE:2022-11-23 01:22:48 4195:4195 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0361766Z STAGE:2022-11-23 01:22:48 4195:4195 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0362453Z STAGE:2022-11-23 01:22:48 4194:4194 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0363163Z STAGE:2022-11-23 01:22:48 4195:4195 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0363878Z STAGE:2022-11-23 01:22:48 4194:4194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0364955Z STAGE:2022-11-23 01:22:48 4194:4194 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:22:48 4195:4195 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0364993Z 
2022-11-23T01:48:21.0365999Z STAGE:2022-11-23 01:22:48 4194:4194 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0366702Z STAGE:2022-11-23 01:22:48 4195:4195 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0367412Z STAGE:2022-11-23 01:22:48 4194:4194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0368116Z STAGE:2022-11-23 01:22:48 4195:4195 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0368341Z ok (4.216s)
2022-11-23T01:48:21.0368370Z 
2022-11-23T01:48:21.0368891Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0369136Z Ran 1 test in 4.216s
2022-11-23T01:48:21.0369166Z 
2022-11-23T01:48:21.0369372Z OK
2022-11-23T01:48:21.0369401Z 
2022-11-23T01:48:21.0369671Z Generating XML reports...
2022-11-23T01:48:21.0370603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012245.xml
2022-11-23T01:48:21.0371521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0371904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0372697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0373074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0373129Z 
2022-11-23T01:48:21.0373345Z Running tests...
2022-11-23T01:48:21.0373889Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0374550Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0375117Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0375573Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4313
2022-11-23T01:48:21.0376032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4314
2022-11-23T01:48:21.0376802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0377177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0377944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0378325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0379159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0379518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0380268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0380661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0381140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0381612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0382384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0383188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0383656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0384115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0384432Z skip: Skipped due to small world size. (4.316s)
2022-11-23T01:48:21.0384468Z 
2022-11-23T01:48:21.0384995Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0385231Z Ran 1 test in 4.317s
2022-11-23T01:48:21.0385259Z 
2022-11-23T01:48:21.0385485Z OK (skipped=1)
2022-11-23T01:48:21.0385512Z 
2022-11-23T01:48:21.0385763Z Generating XML reports...
2022-11-23T01:48:21.0386633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012251.xml
2022-11-23T01:48:21.0387385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0387746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0388518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0388905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0388935Z 
2022-11-23T01:48:21.0389250Z Running tests...
2022-11-23T01:48:21.0389780Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0390420Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0390975Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0391402Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4422
2022-11-23T01:48:21.0391844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4423
2022-11-23T01:48:21.0392610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0392967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0393741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0394124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0394886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0395242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0395994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0396389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0396890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0397473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0398305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0399116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0399607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0400285Z STAGE:2022-11-23 01:23:02 4423:4423 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0401803Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0402058Z   warnings.warn(
2022-11-23T01:48:21.0402512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0403186Z STAGE:2022-11-23 01:23:02 4422:4422 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0404708Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0404962Z   warnings.warn(
2022-11-23T01:48:21.0406668Z STAGE:2022-11-23 01:23:02 4423:4423 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:23:02 4422:4422 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0406709Z 
2022-11-23T01:48:21.0407461Z STAGE:2022-11-23 01:23:02 4422:4422 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:23:02 4423:4423 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0407484Z 
2022-11-23T01:48:21.0407865Z STAGE:2022-11-23 01:23:02 4422:4422 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0408228Z STAGE:2022-11-23 01:23:02 4423:4423 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0408786Z STAGE:2022-11-23 01:23:02 4422:4422 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0409161Z STAGE:2022-11-23 01:23:02 4423:4423 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0409527Z STAGE:2022-11-23 01:23:02 4422:4422 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0409916Z STAGE:2022-11-23 01:23:02 4423:4423 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0410482Z STAGE:2022-11-23 01:23:02 4423:4423 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:23:02 4422:4422 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0410504Z 
2022-11-23T01:48:21.0410876Z STAGE:2022-11-23 01:23:02 4422:4422 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0411243Z STAGE:2022-11-23 01:23:02 4423:4423 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0411627Z STAGE:2022-11-23 01:23:02 4422:4422 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0412004Z STAGE:2022-11-23 01:23:02 4423:4423 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0412147Z ok (4.311s)
2022-11-23T01:48:21.0412168Z 
2022-11-23T01:48:21.0412472Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0412626Z Ran 1 test in 4.312s
2022-11-23T01:48:21.0412646Z 
2022-11-23T01:48:21.0412759Z OK
2022-11-23T01:48:21.0412777Z 
2022-11-23T01:48:21.0412944Z Generating XML reports...
2022-11-23T01:48:21.0413513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012258.xml
2022-11-23T01:48:21.0413941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0414159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0414586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0414818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0414839Z 
2022-11-23T01:48:21.0414987Z Running tests...
2022-11-23T01:48:21.0415273Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0415626Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0415949Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0416207Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4535
2022-11-23T01:48:21.0416456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4536
2022-11-23T01:48:21.0416869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0417087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0417507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0417737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0418121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0418338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0418761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0418990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0419275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0419617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0420062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0420501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0420772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0421030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0421403Z STAGE:2022-11-23 01:23:09 4535:4535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0421761Z STAGE:2022-11-23 01:23:09 4536:4536 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0422554Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0422712Z   warnings.warn(
2022-11-23T01:48:21.0423480Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0423631Z   warnings.warn(
2022-11-23T01:48:21.0424071Z STAGE:2022-11-23 01:23:09 4535:4535 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0424451Z STAGE:2022-11-23 01:23:09 4536:4536 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0424816Z STAGE:2022-11-23 01:23:09 4535:4535 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0425202Z STAGE:2022-11-23 01:23:09 4536:4536 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0425346Z ok (4.215s)
2022-11-23T01:48:21.0425366Z 
2022-11-23T01:48:21.0425671Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0425823Z Ran 1 test in 4.216s
2022-11-23T01:48:21.0425843Z 
2022-11-23T01:48:21.0425974Z OK
2022-11-23T01:48:21.0425993Z 
2022-11-23T01:48:21.0426153Z Generating XML reports...
2022-11-23T01:48:21.0426646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012305.xml
2022-11-23T01:48:21.0427063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0427261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0427685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0427925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0427945Z 
2022-11-23T01:48:21.0428092Z Running tests...
2022-11-23T01:48:21.0428396Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0428750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0429051Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0429307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4648
2022-11-23T01:48:21.0429544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4649
2022-11-23T01:48:21.0429958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0430175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0430664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0430894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0431299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0431517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0431935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0432168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0432435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0432714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0433159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0433600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0433870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0434141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0434511Z STAGE:2022-11-23 01:23:16 4648:4648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0434916Z STAGE:2022-11-23 01:23:16 4649:4649 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0435299Z STAGE:2022-11-23 01:23:16 4648:4648 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0435655Z STAGE:2022-11-23 01:23:16 4649:4649 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0436044Z STAGE:2022-11-23 01:23:16 4648:4648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0436428Z STAGE:2022-11-23 01:23:16 4649:4649 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0436790Z STAGE:2022-11-23 01:23:16 4648:4648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0437148Z STAGE:2022-11-23 01:23:16 4649:4649 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0437513Z STAGE:2022-11-23 01:23:16 4648:4648 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0437884Z STAGE:2022-11-23 01:23:16 4649:4649 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0438262Z STAGE:2022-11-23 01:23:16 4648:4648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0438624Z STAGE:2022-11-23 01:23:16 4649:4649 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0438770Z ok (4.244s)
2022-11-23T01:48:21.0438790Z 
2022-11-23T01:48:21.0439094Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0439248Z Ran 1 test in 4.244s
2022-11-23T01:48:21.0439268Z 
2022-11-23T01:48:21.0439401Z OK
2022-11-23T01:48:21.0439420Z 
2022-11-23T01:48:21.0439583Z Generating XML reports...
2022-11-23T01:48:21.0440077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012312.xml
2022-11-23T01:48:21.0440541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0440756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0441160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0441399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0441473Z 
2022-11-23T01:48:21.0441629Z Running tests...
2022-11-23T01:48:21.0441936Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0442288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0442586Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-11-23T01:48:21.0442607Z 
2022-11-23T01:48:21.0442905Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0443041Z Ran 1 test in 0.002s
2022-11-23T01:48:21.0443060Z 
2022-11-23T01:48:21.0443210Z OK (skipped=1)
2022-11-23T01:48:21.0443230Z 
2022-11-23T01:48:21.0443391Z Generating XML reports...
2022-11-23T01:48:21.0443881Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012319.xml
2022-11-23T01:48:21.0444286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0444502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0444922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0445153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0445173Z 
2022-11-23T01:48:21.0445306Z Running tests...
2022-11-23T01:48:21.0445607Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0446314Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0446635Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-11-23T01:48:21.0446655Z 
2022-11-23T01:48:21.0446962Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0447109Z Ran 1 test in 0.002s
2022-11-23T01:48:21.0447133Z 
2022-11-23T01:48:21.0447279Z OK (skipped=1)
2022-11-23T01:48:21.0447298Z 
2022-11-23T01:48:21.0447461Z Generating XML reports...
2022-11-23T01:48:21.0447948Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012321.xml
2022-11-23T01:48:21.0448342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0448555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0448974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0449203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0449223Z 
2022-11-23T01:48:21.0449368Z Running tests...
2022-11-23T01:48:21.0449666Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0450021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0450321Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0450562Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4827
2022-11-23T01:48:21.0450812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4828
2022-11-23T01:48:21.0451225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0451436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0451857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0452082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0452486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0452773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0453188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0453399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0453681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0453963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0454407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0454842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0455107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0455372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0455649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.0455919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.0456338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0456817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0457198Z STAGE:2022-11-23 01:23:27 4827:4827 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0457553Z STAGE:2022-11-23 01:23:27 4828:4828 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0458127Z STAGE:2022-11-23 01:23:27 4828:4828 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:23:27 4827:4827 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0458151Z 
2022-11-23T01:48:21.0458753Z STAGE:2022-11-23 01:23:27 4828:4828 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:23:27 4827:4827 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0458773Z 
2022-11-23T01:48:21.0459127Z STAGE:2022-11-23 01:23:27 4828:4828 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0459478Z STAGE:2022-11-23 01:23:27 4827:4827 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0459842Z STAGE:2022-11-23 01:23:27 4828:4828 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0460207Z STAGE:2022-11-23 01:23:27 4827:4827 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0460576Z STAGE:2022-11-23 01:23:27 4828:4828 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0460956Z STAGE:2022-11-23 01:23:27 4827:4827 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0461095Z ok (4.321s)
2022-11-23T01:48:21.0461115Z 
2022-11-23T01:48:21.0461417Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0461568Z Ran 1 test in 4.321s
2022-11-23T01:48:21.0461588Z 
2022-11-23T01:48:21.0461717Z OK
2022-11-23T01:48:21.0461737Z 
2022-11-23T01:48:21.0461896Z Generating XML reports...
2022-11-23T01:48:21.0462385Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012323.xml
2022-11-23T01:48:21.0462793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0462991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0463466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0463694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0463713Z 
2022-11-23T01:48:21.0463857Z Running tests...
2022-11-23T01:48:21.0464156Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0464504Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0464798Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0465054Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4946
2022-11-23T01:48:21.0465285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4947
2022-11-23T01:48:21.0465696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0465911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0466329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0466558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0466961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0467171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0467630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0467863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0468129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0468403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0468850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0469283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0469550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0469813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0470008Z skip: Skipped due to small world size. (4.203s)
2022-11-23T01:48:21.0470032Z 
2022-11-23T01:48:21.0470334Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0470483Z Ran 1 test in 4.203s
2022-11-23T01:48:21.0470503Z 
2022-11-23T01:48:21.0470631Z OK (skipped=1)
2022-11-23T01:48:21.0470649Z 
2022-11-23T01:48:21.0470808Z Generating XML reports...
2022-11-23T01:48:21.0471304Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012330.xml
2022-11-23T01:48:21.0471715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0471925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0472342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0472567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0472588Z 
2022-11-23T01:48:21.0472736Z Running tests...
2022-11-23T01:48:21.0473014Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0473365Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0473700Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-11-23T01:48:21.0473770Z 
2022-11-23T01:48:21.0474069Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0474216Z Ran 1 test in 0.002s
2022-11-23T01:48:21.0474236Z 
2022-11-23T01:48:21.0474381Z OK (skipped=1)
2022-11-23T01:48:21.0474400Z 
2022-11-23T01:48:21.0474559Z Generating XML reports...
2022-11-23T01:48:21.0475046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012337.xml
2022-11-23T01:48:21.0475455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0475652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0476066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0476294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0476318Z 
2022-11-23T01:48:21.0476461Z Running tests...
2022-11-23T01:48:21.0476757Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0477105Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0477440Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-11-23T01:48:21.0477460Z 
2022-11-23T01:48:21.0477755Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0477964Z Ran 1 test in 0.002s
2022-11-23T01:48:21.0477986Z 
2022-11-23T01:48:21.0478116Z OK (skipped=1)
2022-11-23T01:48:21.0478135Z 
2022-11-23T01:48:21.0478297Z Generating XML reports...
2022-11-23T01:48:21.0478792Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012339.xml
2022-11-23T01:48:21.0479202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0479412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0479825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0480050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0480070Z 
2022-11-23T01:48:21.0480215Z Running tests...
2022-11-23T01:48:21.0480511Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0480849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0481173Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-11-23T01:48:21.0481193Z 
2022-11-23T01:48:21.0481487Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0481639Z Ran 1 test in 0.002s
2022-11-23T01:48:21.0481658Z 
2022-11-23T01:48:21.0481801Z OK (skipped=1)
2022-11-23T01:48:21.0481821Z 
2022-11-23T01:48:21.0481979Z Generating XML reports...
2022-11-23T01:48:21.0482462Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012342.xml
2022-11-23T01:48:21.0482867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0483078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0483482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0483712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0483731Z 
2022-11-23T01:48:21.0483877Z Running tests...
2022-11-23T01:48:21.0484178Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0484582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0484915Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-11-23T01:48:21.0484935Z 
2022-11-23T01:48:21.0485229Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0485376Z Ran 1 test in 0.002s
2022-11-23T01:48:21.0485396Z 
2022-11-23T01:48:21.0485524Z OK (skipped=1)
2022-11-23T01:48:21.0485560Z 
2022-11-23T01:48:21.0485939Z Generating XML reports...
2022-11-23T01:48:21.0486449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012344.xml
2022-11-23T01:48:21.0486857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0487069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0487489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0487713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0487734Z 
2022-11-23T01:48:21.0487878Z Running tests...
2022-11-23T01:48:21.0488174Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0488506Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0488890Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0489156Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5187
2022-11-23T01:48:21.0489408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5188
2022-11-23T01:48:21.0489819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0490037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0490453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0490678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0491070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0491282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0491701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0491927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0492207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0492485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0492923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0493357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0493624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0493874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0494020Z ok (4.158s)
2022-11-23T01:48:21.0494040Z 
2022-11-23T01:48:21.0494340Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0494491Z Ran 1 test in 4.158s
2022-11-23T01:48:21.0494511Z 
2022-11-23T01:48:21.0494640Z OK
2022-11-23T01:48:21.0494659Z 
2022-11-23T01:48:21.0494818Z Generating XML reports...
2022-11-23T01:48:21.0495379Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012346.xml
2022-11-23T01:48:21.0495786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0495999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0496396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0496625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0496649Z 
2022-11-23T01:48:21.0496791Z Running tests...
2022-11-23T01:48:21.0497091Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0497441Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0497750Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0498010Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5296
2022-11-23T01:48:21.0498266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5297
2022-11-23T01:48:21.0498656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0498870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0499285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0499566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0499975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0500186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0500605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0500832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0501112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0501370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0501812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0502249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0502515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0502779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0503058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.0503329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.0503765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0504196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0504458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:21.0504731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:21.0505162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.0505642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.0505917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:48:21.0506185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:48:21.0506613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T01:48:21.0507045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T01:48:21.0507187Z ok (4.342s)
2022-11-23T01:48:21.0507207Z 
2022-11-23T01:48:21.0507490Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0507642Z Ran 1 test in 4.342s
2022-11-23T01:48:21.0507662Z 
2022-11-23T01:48:21.0507788Z OK
2022-11-23T01:48:21.0507808Z 
2022-11-23T01:48:21.0507973Z Generating XML reports...
2022-11-23T01:48:21.0508465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012353.xml
2022-11-23T01:48:21.0508876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0509089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0509508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0509738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0509802Z 
2022-11-23T01:48:21.0509935Z Running tests...
2022-11-23T01:48:21.0510242Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0510595Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0510895Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.003s)
2022-11-23T01:48:21.0510919Z 
2022-11-23T01:48:21.0511218Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0511367Z Ran 1 test in 0.003s
2022-11-23T01:48:21.0511387Z 
2022-11-23T01:48:21.0511533Z OK (skipped=1)
2022-11-23T01:48:21.0511552Z 
2022-11-23T01:48:21.0511711Z Generating XML reports...
2022-11-23T01:48:21.0512179Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012400.xml
2022-11-23T01:48:21.0512593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0512806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0513222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0513450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0513473Z 
2022-11-23T01:48:21.0513618Z Running tests...
2022-11-23T01:48:21.0513919Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0514268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0514589Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0514830Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5462
2022-11-23T01:48:21.0515087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5463
2022-11-23T01:48:21.0515498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0515711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0516125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0516408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0516810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0517019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0517415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0517642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0517928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0518202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0518640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0519075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0519340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0519601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0519879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.0520181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.0520625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0521056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0521429Z STAGE:2022-11-23 01:24:06 5462:5462 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0522215Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0522366Z   warnings.warn(
2022-11-23T01:48:21.0522727Z STAGE:2022-11-23 01:24:06 5463:5463 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0523492Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0523641Z   warnings.warn(
2022-11-23T01:48:21.0524013Z STAGE:2022-11-23 01:24:06 5462:5462 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0524364Z STAGE:2022-11-23 01:24:06 5463:5463 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0524745Z STAGE:2022-11-23 01:24:06 5462:5462 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0525123Z STAGE:2022-11-23 01:24:06 5463:5463 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0525487Z STAGE:2022-11-23 01:24:06 5463:5463 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0526068Z STAGE:2022-11-23 01:24:06 5462:5462 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0526446Z STAGE:2022-11-23 01:24:06 5462:5462 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0526812Z STAGE:2022-11-23 01:24:06 5463:5463 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0527186Z STAGE:2022-11-23 01:24:06 5462:5462 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0527658Z STAGE:2022-11-23 01:24:06 5463:5463 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0527783Z ok (4.218s)
2022-11-23T01:48:21.0527803Z 
2022-11-23T01:48:21.0528103Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0528251Z Ran 1 test in 4.219s
2022-11-23T01:48:21.0528271Z 
2022-11-23T01:48:21.0528400Z OK
2022-11-23T01:48:21.0528419Z 
2022-11-23T01:48:21.0528577Z Generating XML reports...
2022-11-23T01:48:21.0529069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012402.xml
2022-11-23T01:48:21.0529480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0529693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0530093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0530329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0530348Z 
2022-11-23T01:48:21.0530496Z Running tests...
2022-11-23T01:48:21.0530799Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0531148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0531467Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0531782Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5581
2022-11-23T01:48:21.0532037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5582
2022-11-23T01:48:21.0532446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0532646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0533059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0533288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0533693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0533906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0534325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0534554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0534838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0535113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0535542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0535975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0536240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0536505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0536783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.0537053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.0537485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0537966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0538334Z STAGE:2022-11-23 01:24:13 5581:5581 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0539098Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0539251Z   warnings.warn(
2022-11-23T01:48:21.0539613Z STAGE:2022-11-23 01:24:13 5582:5582 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0540426Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0540584Z   warnings.warn(
2022-11-23T01:48:21.0541162Z STAGE:2022-11-23 01:24:13 5581:5581 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:24:13 5582:5582 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0541183Z 
2022-11-23T01:48:21.0541785Z STAGE:2022-11-23 01:24:13 5582:5582 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:24:13 5581:5581 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0541805Z 
2022-11-23T01:48:21.0542208Z STAGE:2022-11-23 01:24:13 5581:5581 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0542578Z STAGE:2022-11-23 01:24:13 5582:5582 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0542943Z STAGE:2022-11-23 01:24:13 5581:5581 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0543311Z STAGE:2022-11-23 01:24:13 5582:5582 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0543672Z STAGE:2022-11-23 01:24:13 5581:5581 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0544050Z STAGE:2022-11-23 01:24:13 5582:5582 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0544188Z ok (4.349s)
2022-11-23T01:48:21.0544208Z 
2022-11-23T01:48:21.0544508Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0544657Z Ran 1 test in 4.350s
2022-11-23T01:48:21.0544676Z 
2022-11-23T01:48:21.0544805Z OK
2022-11-23T01:48:21.0544827Z 
2022-11-23T01:48:21.0544988Z Generating XML reports...
2022-11-23T01:48:21.0545477Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012409.xml
2022-11-23T01:48:21.0545883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0546083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0546501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0546729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0546749Z 
2022-11-23T01:48:21.0546892Z Running tests...
2022-11-23T01:48:21.0547188Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0547537Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0547869Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0548124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5700
2022-11-23T01:48:21.0548357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5701
2022-11-23T01:48:21.0548852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0549064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0549481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0549709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0550114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0550328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0550744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0550972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0551238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0551517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0551952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0552385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0552650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0552960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0553244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.0553510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.0553949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0554367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0554736Z STAGE:2022-11-23 01:24:20 5700:5700 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0555092Z STAGE:2022-11-23 01:24:20 5701:5701 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0555871Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0556023Z   warnings.warn(
2022-11-23T01:48:21.0556781Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0556931Z   warnings.warn(
2022-11-23T01:48:21.0557508Z STAGE:2022-11-23 01:24:20 5700:5700 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:24:20 5701:5701 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0557529Z 
2022-11-23T01:48:21.0558131Z STAGE:2022-11-23 01:24:20 5700:5700 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:24:20 5701:5701 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0558152Z 
2022-11-23T01:48:21.0558509Z STAGE:2022-11-23 01:24:20 5700:5700 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0558859Z STAGE:2022-11-23 01:24:20 5701:5701 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0559263Z STAGE:2022-11-23 01:24:20 5701:5701 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0559631Z STAGE:2022-11-23 01:24:20 5700:5700 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0560008Z STAGE:2022-11-23 01:24:20 5701:5701 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0560384Z STAGE:2022-11-23 01:24:20 5700:5700 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0560523Z ok (4.231s)
2022-11-23T01:48:21.0560543Z 
2022-11-23T01:48:21.0560848Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0560998Z Ran 1 test in 4.231s
2022-11-23T01:48:21.0561018Z 
2022-11-23T01:48:21.0561146Z OK
2022-11-23T01:48:21.0561165Z 
2022-11-23T01:48:21.0561325Z Generating XML reports...
2022-11-23T01:48:21.0561802Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012416.xml
2022-11-23T01:48:21.0562216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0562429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0562843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0563071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0563090Z 
2022-11-23T01:48:21.0563236Z Running tests...
2022-11-23T01:48:21.0563581Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0563938Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0564244Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0564498Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5819
2022-11-23T01:48:21.0564750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5820
2022-11-23T01:48:21.0565156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0565367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0566003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0566239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0566652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0566863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0567263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0567494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0567777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0568051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0568488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0568921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0569189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0569453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0569716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.0570067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.0570504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0570931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0571295Z STAGE:2022-11-23 01:24:27 5819:5819 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0572071Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0572222Z   warnings.warn(
2022-11-23T01:48:21.0572581Z STAGE:2022-11-23 01:24:27 5820:5820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0573342Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0573490Z   warnings.warn(
2022-11-23T01:48:21.0574066Z STAGE:2022-11-23 01:24:27 5820:5820 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:24:27 5819:5819 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0574087Z 
2022-11-23T01:48:21.0574740Z STAGE:2022-11-23 01:24:27 5819:5819 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:24:27 5820:5820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0574781Z 
2022-11-23T01:48:21.0575132Z STAGE:2022-11-23 01:24:27 5820:5820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0575491Z STAGE:2022-11-23 01:24:27 5819:5819 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0575856Z STAGE:2022-11-23 01:24:27 5820:5820 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0576222Z STAGE:2022-11-23 01:24:27 5819:5819 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0576600Z STAGE:2022-11-23 01:24:27 5820:5820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0576980Z STAGE:2022-11-23 01:24:27 5819:5819 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0577118Z ok (4.351s)
2022-11-23T01:48:21.0577138Z 
2022-11-23T01:48:21.0577436Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0577586Z Ran 1 test in 4.351s
2022-11-23T01:48:21.0577606Z 
2022-11-23T01:48:21.0577718Z OK
2022-11-23T01:48:21.0577736Z 
2022-11-23T01:48:21.0577900Z Generating XML reports...
2022-11-23T01:48:21.0578391Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012423.xml
2022-11-23T01:48:21.0578797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0579010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0579427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0579654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0579677Z 
2022-11-23T01:48:21.0579825Z Running tests...
2022-11-23T01:48:21.0580109Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0580459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0580773Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0581084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5938
2022-11-23T01:48:21.0581328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5939
2022-11-23T01:48:21.0581739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0581950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0582371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0582601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0582990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0583204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0583626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0583851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0584135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0584409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0584846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0585325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0585583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0585848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0586047Z skip: Skipped due to small world size. (4.241s)
2022-11-23T01:48:21.0586067Z 
2022-11-23T01:48:21.0586370Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0586519Z Ran 1 test in 4.241s
2022-11-23T01:48:21.0586538Z 
2022-11-23T01:48:21.0586681Z OK (skipped=1)
2022-11-23T01:48:21.0586701Z 
2022-11-23T01:48:21.0586860Z Generating XML reports...
2022-11-23T01:48:21.0587345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012430.xml
2022-11-23T01:48:21.0587758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0587955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0588369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0588601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0588620Z 
2022-11-23T01:48:21.0588769Z Running tests...
2022-11-23T01:48:21.0589068Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0633766Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0634151Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0634405Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6047
2022-11-23T01:48:21.0634658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6048
2022-11-23T01:48:21.0635101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0635310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0635721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0636096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0636501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0636702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0637115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0637338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0637608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0637883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0638315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0638754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0639018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0639273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0639459Z skip: Skipped due to small world size. (4.238s)
2022-11-23T01:48:21.0639482Z 
2022-11-23T01:48:21.0639782Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0639991Z Ran 1 test in 4.238s
2022-11-23T01:48:21.0640013Z 
2022-11-23T01:48:21.0640190Z OK (skipped=1)
2022-11-23T01:48:21.0640210Z 
2022-11-23T01:48:21.0640367Z Generating XML reports...
2022-11-23T01:48:21.0640869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012437.xml
2022-11-23T01:48:21.0641290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0641506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0641921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0642141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0642161Z 
2022-11-23T01:48:21.0642297Z Running tests...
2022-11-23T01:48:21.0642595Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0642933Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0643247Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0643491Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6156
2022-11-23T01:48:21.0643743Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6157
2022-11-23T01:48:21.0644153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0644366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0644784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0645014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0645400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0645618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0646347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0646666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0646954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0647238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0647681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0648116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0648388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0648640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0648837Z skip: Skipped due to small world size. (4.256s)
2022-11-23T01:48:21.0648857Z 
2022-11-23T01:48:21.0649162Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0649302Z Ran 1 test in 4.257s
2022-11-23T01:48:21.0649321Z 
2022-11-23T01:48:21.0649456Z OK (skipped=1)
2022-11-23T01:48:21.0649476Z 
2022-11-23T01:48:21.0649627Z Generating XML reports...
2022-11-23T01:48:21.0650113Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012443.xml
2022-11-23T01:48:21.0650517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0650789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0651202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0651422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0651442Z 
2022-11-23T01:48:21.0651578Z Running tests...
2022-11-23T01:48:21.0651877Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0652232Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0652548Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0652805Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6265
2022-11-23T01:48:21.0653059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6266
2022-11-23T01:48:21.0653456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0653673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0654090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0654313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0654707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0654908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0655307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0655522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0655797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0656068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0656497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0656919Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0657236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0657490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0657674Z skip: Skipped due to small world size. (4.240s)
2022-11-23T01:48:21.0657694Z 
2022-11-23T01:48:21.0657987Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0658127Z Ran 1 test in 4.241s
2022-11-23T01:48:21.0658147Z 
2022-11-23T01:48:21.0658283Z OK (skipped=1)
2022-11-23T01:48:21.0658302Z 
2022-11-23T01:48:21.0658450Z Generating XML reports...
2022-11-23T01:48:21.0658930Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012450.xml
2022-11-23T01:48:21.0659329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0659534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0659941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0660160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0660179Z 
2022-11-23T01:48:21.0660315Z Running tests...
2022-11-23T01:48:21.0660605Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0660940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0661283Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0661535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6374
2022-11-23T01:48:21.0661776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6375
2022-11-23T01:48:21.0662181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0662382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0662784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0663001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0663393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0663593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0664004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0664221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0664490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0664763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0665190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0665612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0665868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0666120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0666482Z STAGE:2022-11-23 01:25:01 6375:6375 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0666838Z STAGE:2022-11-23 01:25:01 6374:6374 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0667614Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0667809Z   warnings.warn(
2022-11-23T01:48:21.0668571Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0668710Z   warnings.warn(
2022-11-23T01:48:21.0669278Z STAGE:2022-11-23 01:25:01 6374:6374 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:01 6375:6375 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0669299Z 
2022-11-23T01:48:21.0669886Z STAGE:2022-11-23 01:25:01 6374:6374 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:01 6375:6375 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0669910Z 
2022-11-23T01:48:21.0670260Z STAGE:2022-11-23 01:25:01 6375:6375 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0670612Z STAGE:2022-11-23 01:25:01 6374:6374 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0670968Z STAGE:2022-11-23 01:25:01 6374:6374 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0671368Z STAGE:2022-11-23 01:25:01 6375:6375 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0671744Z STAGE:2022-11-23 01:25:01 6374:6374 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0672111Z STAGE:2022-11-23 01:25:01 6375:6375 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0672241Z ok (4.235s)
2022-11-23T01:48:21.0672265Z 
2022-11-23T01:48:21.0672554Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0672694Z Ran 1 test in 4.235s
2022-11-23T01:48:21.0672714Z 
2022-11-23T01:48:21.0672832Z OK
2022-11-23T01:48:21.0672851Z 
2022-11-23T01:48:21.0673002Z Generating XML reports...
2022-11-23T01:48:21.0673475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012457.xml
2022-11-23T01:48:21.0673879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0674085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0674493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0674714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0674734Z 
2022-11-23T01:48:21.0674869Z Running tests...
2022-11-23T01:48:21.0675166Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0675508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0675837Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0676076Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6487
2022-11-23T01:48:21.0676317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6488
2022-11-23T01:48:21.0676719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0676922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0677325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0677544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0677991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0678192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0678586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0678803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0679076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0679350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0679780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0680204Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0680465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0680718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0681484Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0681686Z   warnings.warn(
2022-11-23T01:48:21.0682455Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0682601Z   warnings.warn(
2022-11-23T01:48:21.0682730Z ok (4.249s)
2022-11-23T01:48:21.0682750Z 
2022-11-23T01:48:21.0683045Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0683185Z Ran 1 test in 4.250s
2022-11-23T01:48:21.0683205Z 
2022-11-23T01:48:21.0683322Z OK
2022-11-23T01:48:21.0683341Z 
2022-11-23T01:48:21.0683491Z Generating XML reports...
2022-11-23T01:48:21.0683971Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012504.xml
2022-11-23T01:48:21.0684374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0684573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0684984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0685203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0685226Z 
2022-11-23T01:48:21.0685369Z Running tests...
2022-11-23T01:48:21.0685892Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0686261Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0686568Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0686822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6596
2022-11-23T01:48:21.0687058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6597
2022-11-23T01:48:21.0687473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0687685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0688086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0688384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0688800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0689024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0689433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0689657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0689929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0690213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0690648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0691085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0691354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0691615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0691982Z STAGE:2022-11-23 01:25:14 6596:6596 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0692348Z STAGE:2022-11-23 01:25:14 6597:6597 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0693206Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0693350Z   warnings.warn(
2022-11-23T01:48:21.0694133Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0694277Z   warnings.warn(
2022-11-23T01:48:21.0694640Z STAGE:2022-11-23 01:25:14 6596:6596 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0694988Z STAGE:2022-11-23 01:25:14 6597:6597 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0695360Z STAGE:2022-11-23 01:25:14 6596:6596 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0695730Z STAGE:2022-11-23 01:25:14 6597:6597 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0696082Z STAGE:2022-11-23 01:25:14 6596:6596 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0696434Z STAGE:2022-11-23 01:25:14 6597:6597 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0696783Z STAGE:2022-11-23 01:25:14 6596:6596 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0697136Z STAGE:2022-11-23 01:25:14 6597:6597 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0697504Z STAGE:2022-11-23 01:25:14 6596:6596 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0697872Z STAGE:2022-11-23 01:25:14 6597:6597 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0698003Z ok (4.233s)
2022-11-23T01:48:21.0698028Z 
2022-11-23T01:48:21.0698320Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0698460Z Ran 1 test in 4.233s
2022-11-23T01:48:21.0698480Z 
2022-11-23T01:48:21.0698597Z OK
2022-11-23T01:48:21.0698617Z 
2022-11-23T01:48:21.0698767Z Generating XML reports...
2022-11-23T01:48:21.0699303Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012511.xml
2022-11-23T01:48:21.0699710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0699919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0700332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0700554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0700574Z 
2022-11-23T01:48:21.0700718Z Running tests...
2022-11-23T01:48:21.0701012Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0701354Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0701654Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0701914Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6709
2022-11-23T01:48:21.0702165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6710
2022-11-23T01:48:21.0702576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0702785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0703198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0703466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0703875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0704083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0704478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0704705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0704982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0705257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0705687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0706118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0706433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0706697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0707078Z STAGE:2022-11-23 01:25:21 6710:6710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0707424Z STAGE:2022-11-23 01:25:21 6709:6709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0708190Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0708338Z   warnings.warn(
2022-11-23T01:48:21.0709106Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0709250Z   warnings.warn(
2022-11-23T01:48:21.0709615Z STAGE:2022-11-23 01:25:21 6709:6709 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0710030Z STAGE:2022-11-23 01:25:21 6710:6710 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0710624Z STAGE:2022-11-23 01:25:21 6709:6709 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:21 6710:6710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0710645Z 
2022-11-23T01:48:21.0711201Z STAGE:2022-11-23 01:25:21 6709:6709 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:25:21 6710:6710 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0711222Z 
2022-11-23T01:48:21.0711587Z STAGE:2022-11-23 01:25:21 6710:6710 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0711947Z STAGE:2022-11-23 01:25:21 6709:6709 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0712314Z STAGE:2022-11-23 01:25:21 6710:6710 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0712695Z STAGE:2022-11-23 01:25:21 6709:6709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0712832Z ok (4.151s)
2022-11-23T01:48:21.0712852Z 
2022-11-23T01:48:21.0713151Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0713298Z Ran 1 test in 4.152s
2022-11-23T01:48:21.0713319Z 
2022-11-23T01:48:21.0713443Z OK
2022-11-23T01:48:21.0713462Z 
2022-11-23T01:48:21.0713618Z Generating XML reports...
2022-11-23T01:48:21.0714153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012517.xml
2022-11-23T01:48:21.0714556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0714768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0715179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0715410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0715430Z 
2022-11-23T01:48:21.0715572Z Running tests...
2022-11-23T01:48:21.0715875Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0716223Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0716528Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0716785Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6822
2022-11-23T01:48:21.0717023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6823
2022-11-23T01:48:21.0717431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0717644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0718054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0718282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0718675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0718880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0719288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0719495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0719776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0720056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0720545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0720972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0721234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0721596Z STAGE:2022-11-23 01:25:28 6823:6823 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0722367Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0722520Z   warnings.warn(
2022-11-23T01:48:21.0722781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0723140Z STAGE:2022-11-23 01:25:28 6822:6822 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0723905Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:48:21.0724051Z   warnings.warn(
2022-11-23T01:48:21.0724416Z STAGE:2022-11-23 01:25:28 6822:6822 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0724816Z STAGE:2022-11-23 01:25:28 6823:6823 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0725200Z STAGE:2022-11-23 01:25:28 6822:6822 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0725575Z STAGE:2022-11-23 01:25:28 6823:6823 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0726157Z STAGE:2022-11-23 01:25:28 6823:6823 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0726515Z STAGE:2022-11-23 01:25:28 6822:6822 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0727075Z STAGE:2022-11-23 01:25:28 6822:6822 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:28 6823:6823 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0727109Z 
2022-11-23T01:48:21.0727695Z STAGE:2022-11-23 01:25:28 6823:6823 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:28 6822:6822 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0727726Z 
2022-11-23T01:48:21.0727849Z ok (4.341s)
2022-11-23T01:48:21.0727868Z 
2022-11-23T01:48:21.0728165Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0728316Z Ran 1 test in 4.341s
2022-11-23T01:48:21.0728335Z 
2022-11-23T01:48:21.0728459Z OK
2022-11-23T01:48:21.0728477Z 
2022-11-23T01:48:21.0728642Z Generating XML reports...
2022-11-23T01:48:21.0729129Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012524.xml
2022-11-23T01:48:21.0729534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0729744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0730152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0730381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0730401Z 
2022-11-23T01:48:21.0730541Z Running tests...
2022-11-23T01:48:21.0730835Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0731288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0731610Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0731860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6935
2022-11-23T01:48:21.0732119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6936
2022-11-23T01:48:21.0732522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0732726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0733141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0733372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0733765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0733977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0734393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0734614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0734892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0735157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0735691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0736134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0736397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0736671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0736806Z ok (4.304s)
2022-11-23T01:48:21.0736826Z 
2022-11-23T01:48:21.0737122Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0737274Z Ran 1 test in 4.304s
2022-11-23T01:48:21.0737294Z 
2022-11-23T01:48:21.0737417Z OK
2022-11-23T01:48:21.0737436Z 
2022-11-23T01:48:21.0737581Z Generating XML reports...
2022-11-23T01:48:21.0738072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012531.xml
2022-11-23T01:48:21.0738485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0738697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0739106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0739339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0739359Z 
2022-11-23T01:48:21.0739501Z Running tests...
2022-11-23T01:48:21.0739799Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0740131Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0740478Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0740735Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7044
2022-11-23T01:48:21.0740984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7045
2022-11-23T01:48:21.0741397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0741607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0742077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0742307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0742701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0742895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0743296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0743520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0743797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0744081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0744514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0744943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0745204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0745466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0745775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.0746061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.0746496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0746926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0747299Z STAGE:2022-11-23 01:25:42 7045:7045 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0747657Z STAGE:2022-11-23 01:25:42 7044:7044 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0748228Z STAGE:2022-11-23 01:25:42 7045:7045 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:42 7044:7044 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0748249Z 
2022-11-23T01:48:21.0748856Z STAGE:2022-11-23 01:25:42 7045:7045 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:42 7044:7044 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0748876Z 
2022-11-23T01:48:21.0749228Z STAGE:2022-11-23 01:25:42 7045:7045 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0749583Z STAGE:2022-11-23 01:25:42 7044:7044 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0749934Z STAGE:2022-11-23 01:25:42 7045:7045 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0750300Z STAGE:2022-11-23 01:25:42 7044:7044 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0750671Z STAGE:2022-11-23 01:25:42 7045:7045 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0751039Z STAGE:2022-11-23 01:25:42 7044:7044 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0751181Z ok (4.411s)
2022-11-23T01:48:21.0751205Z 
2022-11-23T01:48:21.0751500Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0751648Z Ran 1 test in 4.411s
2022-11-23T01:48:21.0751668Z 
2022-11-23T01:48:21.0751800Z OK
2022-11-23T01:48:21.0751819Z 
2022-11-23T01:48:21.0751976Z Generating XML reports...
2022-11-23T01:48:21.0752500Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012538.xml
2022-11-23T01:48:21.0752904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0753118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0753531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0753752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0753772Z 
2022-11-23T01:48:21.0753924Z Running tests...
2022-11-23T01:48:21.0754218Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0754565Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0754854Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0755123Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7163
2022-11-23T01:48:21.0755370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7164
2022-11-23T01:48:21.0755773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0755980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0756395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0756683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0757089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0757299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0757700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0757936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0758213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0758488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0758929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0759360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0759623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0759895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0760170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.0760433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.0760865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0761296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0761661Z STAGE:2022-11-23 01:25:49 7163:7163 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0762022Z STAGE:2022-11-23 01:25:49 7164:7164 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0762392Z STAGE:2022-11-23 01:25:49 7163:7163 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0762748Z STAGE:2022-11-23 01:25:49 7164:7164 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0763189Z STAGE:2022-11-23 01:25:49 7163:7163 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0763569Z STAGE:2022-11-23 01:25:49 7164:7164 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0763912Z STAGE:2022-11-23 01:25:49 7164:7164 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0764268Z STAGE:2022-11-23 01:25:49 7163:7163 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0764837Z STAGE:2022-11-23 01:25:49 7164:7164 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:49 7163:7163 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0764858Z 
2022-11-23T01:48:21.0765237Z STAGE:2022-11-23 01:25:49 7164:7164 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0765607Z STAGE:2022-11-23 01:25:49 7163:7163 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0765962Z ok (4.344s)
2022-11-23T01:48:21.0765984Z 
2022-11-23T01:48:21.0766297Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0766443Z Ran 1 test in 4.344s
2022-11-23T01:48:21.0766463Z 
2022-11-23T01:48:21.0766590Z OK
2022-11-23T01:48:21.0766609Z 
2022-11-23T01:48:21.0766755Z Generating XML reports...
2022-11-23T01:48:21.0767248Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012545.xml
2022-11-23T01:48:21.0767723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0767943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0768366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0768593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0768617Z 
2022-11-23T01:48:21.0768760Z Running tests...
2022-11-23T01:48:21.0769062Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0769397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0769709Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0769959Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7282
2022-11-23T01:48:21.0770219Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7283
2022-11-23T01:48:21.0770627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0770836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0771261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0771489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0771884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0772078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0772490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0772709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0772988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0773270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0773711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0774213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0774475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0774744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0775005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.0775280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.0775719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0776151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0776521Z STAGE:2022-11-23 01:25:56 7282:7282 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0776878Z STAGE:2022-11-23 01:25:56 7283:7283 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0777246Z STAGE:2022-11-23 01:25:56 7282:7282 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0777608Z STAGE:2022-11-23 01:25:56 7283:7283 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0777978Z STAGE:2022-11-23 01:25:56 7282:7282 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0778386Z STAGE:2022-11-23 01:25:56 7283:7283 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0778750Z STAGE:2022-11-23 01:25:56 7282:7282 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0779109Z STAGE:2022-11-23 01:25:56 7283:7283 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0779472Z STAGE:2022-11-23 01:25:56 7282:7282 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0779833Z STAGE:2022-11-23 01:25:56 7283:7283 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0780209Z STAGE:2022-11-23 01:25:56 7282:7282 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0780577Z STAGE:2022-11-23 01:25:56 7283:7283 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0780714Z ok (4.322s)
2022-11-23T01:48:21.0780734Z 
2022-11-23T01:48:21.0781028Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0781168Z Ran 1 test in 4.322s
2022-11-23T01:48:21.0781188Z 
2022-11-23T01:48:21.0781315Z OK
2022-11-23T01:48:21.0781335Z 
2022-11-23T01:48:21.0781500Z Generating XML reports...
2022-11-23T01:48:21.0781983Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012552.xml
2022-11-23T01:48:21.0782392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0782609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0783021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0783243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0783264Z 
2022-11-23T01:48:21.0783392Z Running tests...
2022-11-23T01:48:21.0783692Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0784044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0784346Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0784604Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7401
2022-11-23T01:48:21.0784916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7402
2022-11-23T01:48:21.0785321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0785528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0785932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0786128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0786550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0786775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0787188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0787420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0787697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0787975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0788406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0788821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0789134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0789409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0789680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.0789959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.0790400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0790822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.0791184Z STAGE:2022-11-23 01:26:02 7402:7402 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0791539Z STAGE:2022-11-23 01:26:02 7401:7401 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0792114Z STAGE:2022-11-23 01:26:02 7401:7401 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:26:02 7402:7402 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0792135Z 
2022-11-23T01:48:21.0792716Z STAGE:2022-11-23 01:26:02 7402:7402 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:26:02 7401:7401 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0792760Z 
2022-11-23T01:48:21.0793105Z STAGE:2022-11-23 01:26:02 7402:7402 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0793462Z STAGE:2022-11-23 01:26:02 7401:7401 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0793828Z STAGE:2022-11-23 01:26:02 7401:7401 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0794198Z STAGE:2022-11-23 01:26:02 7402:7402 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0794574Z STAGE:2022-11-23 01:26:02 7402:7402 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0794948Z STAGE:2022-11-23 01:26:02 7401:7401 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0795082Z ok (4.260s)
2022-11-23T01:48:21.0795101Z 
2022-11-23T01:48:21.0795454Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0795588Z Ran 1 test in 4.260s
2022-11-23T01:48:21.0795628Z 
2022-11-23T01:48:21.0795740Z OK
2022-11-23T01:48:21.0795759Z 
2022-11-23T01:48:21.0795914Z Generating XML reports...
2022-11-23T01:48:21.0796398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012558.xml
2022-11-23T01:48:21.0796810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0797024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0797437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0797668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0797688Z 
2022-11-23T01:48:21.0797828Z Running tests...
2022-11-23T01:48:21.0798120Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0798471Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0798765Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0799016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7520
2022-11-23T01:48:21.0799268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7521
2022-11-23T01:48:21.0799726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0799940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0800353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0800582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0800970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0801176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0801582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0801800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0802076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0802362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0802796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0803223Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0803480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0803748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0803947Z skip: Skipped due to small world size. (4.138s)
2022-11-23T01:48:21.0803967Z 
2022-11-23T01:48:21.0804266Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0804412Z Ran 1 test in 4.138s
2022-11-23T01:48:21.0804432Z 
2022-11-23T01:48:21.0804577Z OK (skipped=1)
2022-11-23T01:48:21.0804597Z 
2022-11-23T01:48:21.0804756Z Generating XML reports...
2022-11-23T01:48:21.0805242Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012605.xml
2022-11-23T01:48:21.0805653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0806138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0806562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0806786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0806806Z 
2022-11-23T01:48:21.0806948Z Running tests...
2022-11-23T01:48:21.0807246Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0807596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0807893Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0808143Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7629
2022-11-23T01:48:21.0808379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7630
2022-11-23T01:48:21.0808795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0809008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0809421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0809650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0810047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0810252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0810722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0810956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0811222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0811511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0811950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0812377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0812644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0812905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0813100Z skip: Skipped due to small world size. (4.227s)
2022-11-23T01:48:21.0813120Z 
2022-11-23T01:48:21.0813426Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0813570Z Ran 1 test in 4.227s
2022-11-23T01:48:21.0813590Z 
2022-11-23T01:48:21.0813722Z OK (skipped=1)
2022-11-23T01:48:21.0813742Z 
2022-11-23T01:48:21.0813898Z Generating XML reports...
2022-11-23T01:48:21.0814381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012612.xml
2022-11-23T01:48:21.0814782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0814998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0815410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0815638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0815657Z 
2022-11-23T01:48:21.0815807Z Running tests...
2022-11-23T01:48:21.0816089Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0816437Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0816796Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0817051Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7738
2022-11-23T01:48:21.0817299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7739
2022-11-23T01:48:21.0817704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0817910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0818324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0818557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0818940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0819152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0819561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0819787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0820065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0820340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0820822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0821260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0821523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0821775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0821974Z skip: Skipped due to small world size. (4.291s)
2022-11-23T01:48:21.0821994Z 
2022-11-23T01:48:21.0822302Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0822452Z Ran 1 test in 4.291s
2022-11-23T01:48:21.0822472Z 
2022-11-23T01:48:21.0822620Z OK (skipped=1)
2022-11-23T01:48:21.0822640Z 
2022-11-23T01:48:21.0822796Z Generating XML reports...
2022-11-23T01:48:21.0823281Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012619.xml
2022-11-23T01:48:21.0823699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0823894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0824317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0824552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0824571Z 
2022-11-23T01:48:21.0824719Z Running tests...
2022-11-23T01:48:21.0825020Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0825371Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0825673Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0825932Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7847
2022-11-23T01:48:21.0826191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7848
2022-11-23T01:48:21.0826584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0826802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0827278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0827509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0827908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0828122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0828530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0828763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0829032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0829308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0829749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0830181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0830446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0830710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0830908Z skip: Skipped due to small world size. (4.212s)
2022-11-23T01:48:21.0830927Z 
2022-11-23T01:48:21.0831286Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0831444Z Ran 1 test in 4.212s
2022-11-23T01:48:21.0831464Z 
2022-11-23T01:48:21.0831591Z OK (skipped=1)
2022-11-23T01:48:21.0831610Z 
2022-11-23T01:48:21.0831775Z Generating XML reports...
2022-11-23T01:48:21.0832262Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012626.xml
2022-11-23T01:48:21.0832670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0832885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0833292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0833518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0833538Z 
2022-11-23T01:48:21.0833683Z Running tests...
2022-11-23T01:48:21.0833982Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0834316Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0834601Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0834852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7956
2022-11-23T01:48:21.0835104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7957
2022-11-23T01:48:21.0835506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0835713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0836122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0836344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0836733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0836940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0837352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0837632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0837912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0838189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0838624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0839057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0839321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0839570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0839936Z STAGE:2022-11-23 01:26:36 7956:7956 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0840337Z STAGE:2022-11-23 01:26:36 7957:7957 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0840909Z STAGE:2022-11-23 01:26:36 7956:7956 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:26:36 7957:7957 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0840930Z 
2022-11-23T01:48:21.0841534Z STAGE:2022-11-23 01:26:36 7956:7956 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:26:36 7957:7957 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0841554Z 
2022-11-23T01:48:21.0841962Z STAGE:2022-11-23 01:26:36 7956:7956 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0842329Z STAGE:2022-11-23 01:26:36 7957:7957 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0842908Z STAGE:2022-11-23 01:26:36 7957:7957 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:26:36 7956:7956 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0842932Z 
2022-11-23T01:48:21.0843525Z STAGE:2022-11-23 01:26:36 7957:7957 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:26:36 7956:7956 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0843545Z 
2022-11-23T01:48:21.0843685Z ok (4.256s)
2022-11-23T01:48:21.0843705Z 
2022-11-23T01:48:21.0844007Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0844160Z Ran 1 test in 4.256s
2022-11-23T01:48:21.0844184Z 
2022-11-23T01:48:21.0844297Z OK
2022-11-23T01:48:21.0844316Z 
2022-11-23T01:48:21.0844482Z Generating XML reports...
2022-11-23T01:48:21.0844976Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012632.xml
2022-11-23T01:48:21.0845387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0845606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0846268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0846501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0846522Z 
2022-11-23T01:48:21.0846672Z Running tests...
2022-11-23T01:48:21.0846955Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0847314Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0847604Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0847862Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8069
2022-11-23T01:48:21.0848117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8070
2022-11-23T01:48:21.0848613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0848828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0849248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0849480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0849866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0850086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0850503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0850734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0851023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0851306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0851748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0852187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0852457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0852870Z STAGE:2022-11-23 01:26:43 8070:8070 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0853143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0853509Z STAGE:2022-11-23 01:26:43 8069:8069 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0854087Z STAGE:2022-11-23 01:26:43 8069:8069 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:26:43 8070:8070 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0854108Z 
2022-11-23T01:48:21.0854708Z STAGE:2022-11-23 01:26:43 8069:8069 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:26:43 8070:8070 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0854728Z 
2022-11-23T01:48:21.0855090Z STAGE:2022-11-23 01:26:43 8069:8069 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0855450Z STAGE:2022-11-23 01:26:43 8070:8070 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0855820Z STAGE:2022-11-23 01:26:43 8069:8069 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0856185Z STAGE:2022-11-23 01:26:43 8070:8070 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0856574Z STAGE:2022-11-23 01:26:43 8069:8069 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0856938Z STAGE:2022-11-23 01:26:43 8070:8070 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0857081Z ok (4.353s)
2022-11-23T01:48:21.0857101Z 
2022-11-23T01:48:21.0857403Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0857555Z Ran 1 test in 4.353s
2022-11-23T01:48:21.0857575Z 
2022-11-23T01:48:21.0857705Z OK
2022-11-23T01:48:21.0857724Z 
2022-11-23T01:48:21.0857886Z Generating XML reports...
2022-11-23T01:48:21.0858383Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012639.xml
2022-11-23T01:48:21.0858797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0858994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0859471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0859703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0859722Z 
2022-11-23T01:48:21.0859874Z Running tests...
2022-11-23T01:48:21.0860174Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0860531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0860837Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0861094Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8182
2022-11-23T01:48:21.0861349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8183
2022-11-23T01:48:21.0861742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0861962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0862384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0862616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0863022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0863235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0863689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0863925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0864193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0864486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0864933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0865369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0865636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0865904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0866276Z STAGE:2022-11-23 01:26:51 8183:8183 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0867084Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:48:21.0867245Z   warnings.warn(
2022-11-23T01:48:21.0867617Z STAGE:2022-11-23 01:26:51 8182:8182 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0868401Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:48:21.0868554Z   warnings.warn(
2022-11-23T01:48:21.0868931Z STAGE:2022-11-23 01:26:51 8182:8182 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0869303Z STAGE:2022-11-23 01:26:51 8183:8183 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0869681Z STAGE:2022-11-23 01:26:51 8182:8182 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0870116Z STAGE:2022-11-23 01:26:51 8183:8183 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0870476Z STAGE:2022-11-23 01:26:51 8183:8183 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0870838Z STAGE:2022-11-23 01:26:51 8182:8182 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0871413Z STAGE:2022-11-23 01:26:51 8183:8183 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:26:51 8182:8182 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0871434Z 
2022-11-23T01:48:21.0871816Z STAGE:2022-11-23 01:26:51 8182:8182 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0872179Z STAGE:2022-11-23 01:26:51 8183:8183 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0872323Z ok (5.308s)
2022-11-23T01:48:21.0872343Z 
2022-11-23T01:48:21.0872651Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0872804Z Ran 1 test in 5.308s
2022-11-23T01:48:21.0872823Z 
2022-11-23T01:48:21.0872954Z OK
2022-11-23T01:48:21.0872973Z 
2022-11-23T01:48:21.0873139Z Generating XML reports...
2022-11-23T01:48:21.0873625Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012646.xml
2022-11-23T01:48:21.0874034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0874231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0874697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0874935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0874955Z 
2022-11-23T01:48:21.0875104Z Running tests...
2022-11-23T01:48:21.0875414Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0875767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0876085Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0876346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8297
2022-11-23T01:48:21.0876601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8298
2022-11-23T01:48:21.0876998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0877212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0877630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0877861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0878265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0878478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0878886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0879113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0879379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0879666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0880110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0880541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0880868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0881135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0881508Z STAGE:2022-11-23 01:26:59 8298:8298 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0882313Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:48:21.0882469Z   warnings.warn(
2022-11-23T01:48:21.0882835Z STAGE:2022-11-23 01:26:59 8297:8297 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0883618Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:48:21.0883775Z   warnings.warn(
2022-11-23T01:48:21.0884152Z STAGE:2022-11-23 01:26:59 8297:8297 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0884515Z STAGE:2022-11-23 01:26:59 8298:8298 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0884896Z STAGE:2022-11-23 01:26:59 8297:8297 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0885321Z STAGE:2022-11-23 01:26:59 8298:8298 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0885908Z STAGE:2022-11-23 01:26:59 8297:8297 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0886286Z STAGE:2022-11-23 01:26:59 8298:8298 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0886662Z STAGE:2022-11-23 01:26:59 8297:8297 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0887011Z STAGE:2022-11-23 01:26:59 8298:8298 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0887395Z STAGE:2022-11-23 01:26:59 8297:8297 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0887772Z STAGE:2022-11-23 01:26:59 8298:8298 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0887915Z ok (5.453s)
2022-11-23T01:48:21.0887936Z 
2022-11-23T01:48:21.0888247Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0888400Z Ran 1 test in 5.453s
2022-11-23T01:48:21.0888420Z 
2022-11-23T01:48:21.0888550Z OK
2022-11-23T01:48:21.0888569Z 
2022-11-23T01:48:21.0888731Z Generating XML reports...
2022-11-23T01:48:21.0889202Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012654.xml
2022-11-23T01:48:21.0889622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0889842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0890266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0890502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0890522Z 
2022-11-23T01:48:21.0890672Z Running tests...
2022-11-23T01:48:21.0890978Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0891332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0891633Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0891871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8412
2022-11-23T01:48:21.0892216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8413
2022-11-23T01:48:21.0892631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0892850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0893272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0893501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0893906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0894123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0894519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0894756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0895042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0895324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0895765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0896197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0896532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0896918Z STAGE:2022-11-23 01:27:06 8413:8413 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0897187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0897539Z STAGE:2022-11-23 01:27:06 8412:8412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0897913Z STAGE:2022-11-23 01:27:06 8412:8412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0898275Z STAGE:2022-11-23 01:27:06 8413:8413 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0898657Z STAGE:2022-11-23 01:27:06 8412:8412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0899032Z STAGE:2022-11-23 01:27:06 8413:8413 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0899391Z STAGE:2022-11-23 01:27:06 8412:8412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0899750Z STAGE:2022-11-23 01:27:06 8413:8413 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0900120Z STAGE:2022-11-23 01:27:06 8412:8412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0900490Z STAGE:2022-11-23 01:27:06 8413:8413 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0900851Z STAGE:2022-11-23 01:27:06 8412:8412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0901234Z STAGE:2022-11-23 01:27:06 8413:8413 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0901380Z ok (4.339s)
2022-11-23T01:48:21.0901401Z 
2022-11-23T01:48:21.0901702Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0901854Z Ran 1 test in 4.339s
2022-11-23T01:48:21.0901874Z 
2022-11-23T01:48:21.0902003Z OK
2022-11-23T01:48:21.0902026Z 
2022-11-23T01:48:21.0902188Z Generating XML reports...
2022-11-23T01:48:21.0902684Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012702.xml
2022-11-23T01:48:21.0903095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0903390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0903811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0904042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0904062Z 
2022-11-23T01:48:21.0904209Z Running tests...
2022-11-23T01:48:21.0904509Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0904866Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0905170Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0905429Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8525
2022-11-23T01:48:21.0905665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8526
2022-11-23T01:48:21.0906087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0906302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0906725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0906954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0907358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0907636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0908062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0908293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0908564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0908839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0909285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0909719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0909987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0910257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0910399Z ok (5.244s)
2022-11-23T01:48:21.0910420Z 
2022-11-23T01:48:21.0910724Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0910858Z Ran 1 test in 5.245s
2022-11-23T01:48:21.0910901Z 
2022-11-23T01:48:21.0911012Z OK
2022-11-23T01:48:21.0911031Z 
2022-11-23T01:48:21.0911234Z Generating XML reports...
2022-11-23T01:48:21.0911730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012709.xml
2022-11-23T01:48:21.0912143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0912358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0912779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0913019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0913039Z 
2022-11-23T01:48:21.0913188Z Running tests...
2022-11-23T01:48:21.0913471Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0913823Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0914169Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0914426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8636
2022-11-23T01:48:21.0914682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8637
2022-11-23T01:48:21.0915094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0915307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0915729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0915942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0916347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0916566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0916980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0917206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0917490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0917771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0918254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0918700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0918953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0919335Z STAGE:2022-11-23 01:27:20 8637:8637 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0919601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0919966Z STAGE:2022-11-23 01:27:20 8636:8636 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0920335Z STAGE:2022-11-23 01:27:20 8636:8636 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0920700Z STAGE:2022-11-23 01:27:20 8637:8637 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0921082Z STAGE:2022-11-23 01:27:20 8636:8636 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0921454Z STAGE:2022-11-23 01:27:20 8637:8637 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0921810Z STAGE:2022-11-23 01:27:20 8636:8636 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0922158Z STAGE:2022-11-23 01:27:20 8637:8637 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0922737Z STAGE:2022-11-23 01:27:20 8637:8637 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:27:20 8636:8636 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0922758Z 
2022-11-23T01:48:21.0923138Z STAGE:2022-11-23 01:27:20 8636:8636 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0923514Z STAGE:2022-11-23 01:27:20 8637:8637 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0923656Z ok (4.216s)
2022-11-23T01:48:21.0923681Z 
2022-11-23T01:48:21.0923988Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0924139Z Ran 1 test in 4.217s
2022-11-23T01:48:21.0924159Z 
2022-11-23T01:48:21.0924290Z OK
2022-11-23T01:48:21.0924309Z 
2022-11-23T01:48:21.0924473Z Generating XML reports...
2022-11-23T01:48:21.0925010Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012717.xml
2022-11-23T01:48:21.0925424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0925640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0926294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0926529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0926549Z 
2022-11-23T01:48:21.0926706Z Running tests...
2022-11-23T01:48:21.0927006Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0927361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0927662Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0927905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8749
2022-11-23T01:48:21.0928158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8750
2022-11-23T01:48:21.0928569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0928784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0929199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0929501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0929918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0930134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0930527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0930763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0931052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0931334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0931776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0932214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0932483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0932749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0933121Z STAGE:2022-11-23 01:27:27 8749:8749 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0933469Z STAGE:2022-11-23 01:27:27 8750:8750 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0934047Z STAGE:2022-11-23 01:27:27 8749:8749 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:27:27 8750:8750 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0934068Z 
2022-11-23T01:48:21.0934672Z STAGE:2022-11-23 01:27:27 8749:8749 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:27:27 8750:8750 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0934693Z 
2022-11-23T01:48:21.0935252Z STAGE:2022-11-23 01:27:27 8749:8749 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:27:27 8750:8750 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0935272Z 
2022-11-23T01:48:21.0935716Z STAGE:2022-11-23 01:27:27 8749:8749 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0936082Z STAGE:2022-11-23 01:27:27 8750:8750 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0936460Z STAGE:2022-11-23 01:27:27 8749:8749 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0936839Z STAGE:2022-11-23 01:27:27 8750:8750 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0936982Z ok (4.249s)
2022-11-23T01:48:21.0937003Z 
2022-11-23T01:48:21.0937308Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0937461Z Ran 1 test in 4.249s
2022-11-23T01:48:21.0937481Z 
2022-11-23T01:48:21.0937591Z OK
2022-11-23T01:48:21.0937610Z 
2022-11-23T01:48:21.0937774Z Generating XML reports...
2022-11-23T01:48:21.0938264Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012723.xml
2022-11-23T01:48:21.0938679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0938893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0939316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0939549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0939568Z 
2022-11-23T01:48:21.0939715Z Running tests...
2022-11-23T01:48:21.0940017Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0940441Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0940757Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0941016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8862
2022-11-23T01:48:21.0941275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8863
2022-11-23T01:48:21.0941692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0941908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0942329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0942558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0942947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0943160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0943571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0943804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0944090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0944381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0944820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0945254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0945526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0945776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0946151Z STAGE:2022-11-23 01:27:34 8863:8863 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0946571Z STAGE:2022-11-23 01:27:34 8862:8862 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0946943Z STAGE:2022-11-23 01:27:34 8862:8862 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0947310Z STAGE:2022-11-23 01:27:34 8863:8863 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0947689Z STAGE:2022-11-23 01:27:34 8862:8862 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0948062Z STAGE:2022-11-23 01:27:34 8863:8863 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0948424Z STAGE:2022-11-23 01:27:34 8863:8863 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0948788Z STAGE:2022-11-23 01:27:34 8862:8862 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0949346Z STAGE:2022-11-23 01:27:34 8863:8863 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:27:34 8862:8862 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0949390Z 
2022-11-23T01:48:21.0949970Z STAGE:2022-11-23 01:27:34 8863:8863 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:27:34 8862:8862 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0950008Z 
2022-11-23T01:48:21.0950131Z ok (4.257s)
2022-11-23T01:48:21.0950150Z 
2022-11-23T01:48:21.0950455Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0950608Z Ran 1 test in 4.258s
2022-11-23T01:48:21.0950628Z 
2022-11-23T01:48:21.0950756Z OK
2022-11-23T01:48:21.0950821Z 
2022-11-23T01:48:21.0950988Z Generating XML reports...
2022-11-23T01:48:21.0951484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012730.xml
2022-11-23T01:48:21.0951898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0952119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0952522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0952752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0952772Z 
2022-11-23T01:48:21.0952918Z Running tests...
2022-11-23T01:48:21.0953219Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0953572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0953874Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0954131Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8975
2022-11-23T01:48:21.0954383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8976
2022-11-23T01:48:21.0954783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0954999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0955418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0955649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0956053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0956271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0956681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0956908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0957194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0957517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0957958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0958394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0958662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0958933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0959303Z STAGE:2022-11-23 01:27:42 8976:8976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0959665Z STAGE:2022-11-23 01:27:42 8975:8975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0960041Z STAGE:2022-11-23 01:27:42 8975:8975 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0960407Z STAGE:2022-11-23 01:27:42 8976:8976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0960767Z STAGE:2022-11-23 01:27:42 8975:8975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0961147Z STAGE:2022-11-23 01:27:42 8976:8976 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0961507Z STAGE:2022-11-23 01:27:42 8975:8975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0961911Z STAGE:2022-11-23 01:27:42 8976:8976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0962496Z STAGE:2022-11-23 01:27:43 8975:8975 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:27:43 8976:8976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0962521Z 
2022-11-23T01:48:21.0963122Z STAGE:2022-11-23 01:27:43 8975:8975 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:27:43 8976:8976 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0963143Z 
2022-11-23T01:48:21.0963701Z STAGE:2022-11-23 01:27:43 8975:8975 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:27:43 8976:8976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0963721Z 
2022-11-23T01:48:21.0964088Z STAGE:2022-11-23 01:27:43 8975:8975 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0964451Z STAGE:2022-11-23 01:27:43 8976:8976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0965050Z STAGE:2022-11-23 01:27:43 8975:8975 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:27:43 8976:8976 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0965074Z 
2022-11-23T01:48:21.0965219Z ok (6.136s)
2022-11-23T01:48:21.0965238Z 
2022-11-23T01:48:21.0965543Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0965891Z Ran 1 test in 6.136s
2022-11-23T01:48:21.0965912Z 
2022-11-23T01:48:21.0966052Z OK
2022-11-23T01:48:21.0966071Z 
2022-11-23T01:48:21.0966242Z Generating XML reports...
2022-11-23T01:48:21.0966744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012737.xml
2022-11-23T01:48:21.0967161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0967377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0967799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0968030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0968123Z 
2022-11-23T01:48:21.0968257Z Running tests...
2022-11-23T01:48:21.0968564Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0968923Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0969232Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0969487Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9091
2022-11-23T01:48:21.0969741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9092
2022-11-23T01:48:21.0970156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0970373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0970780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0970980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0971398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0971628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0972042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0972268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0972613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0972906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0973351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0973777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0974049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0974317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0974690Z STAGE:2022-11-23 01:27:50 9091:9091 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0975053Z STAGE:2022-11-23 01:27:50 9092:9092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0975630Z STAGE:2022-11-23 01:27:50 9092:9092 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:27:50 9091:9091 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0975651Z 
2022-11-23T01:48:21.0976250Z STAGE:2022-11-23 01:27:50 9091:9091 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:27:50 9092:9092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0976274Z 
2022-11-23T01:48:21.0976638Z STAGE:2022-11-23 01:27:50 9091:9091 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0976997Z STAGE:2022-11-23 01:27:50 9092:9092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0977571Z STAGE:2022-11-23 01:27:51 9091:9091 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:27:51 9092:9092 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0977592Z 
2022-11-23T01:48:21.0978190Z STAGE:2022-11-23 01:27:51 9091:9091 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:27:51 9092:9092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0978211Z 
2022-11-23T01:48:21.0978768Z STAGE:2022-11-23 01:27:51 9091:9091 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:27:51 9092:9092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0978836Z 
2022-11-23T01:48:21.0979216Z STAGE:2022-11-23 01:27:51 9091:9091 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0979566Z STAGE:2022-11-23 01:27:51 9092:9092 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0979943Z STAGE:2022-11-23 01:27:51 9091:9091 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0980325Z STAGE:2022-11-23 01:27:51 9092:9092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0980471Z ok (6.108s)
2022-11-23T01:48:21.0980491Z 
2022-11-23T01:48:21.0980794Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0980946Z Ran 1 test in 6.109s
2022-11-23T01:48:21.0980966Z 
2022-11-23T01:48:21.0981095Z OK
2022-11-23T01:48:21.0981114Z 
2022-11-23T01:48:21.0981282Z Generating XML reports...
2022-11-23T01:48:21.0981779Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012746.xml
2022-11-23T01:48:21.0982171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0982387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0982808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0983041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0983118Z 
2022-11-23T01:48:21.0983273Z Running tests...
2022-11-23T01:48:21.0983578Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0983936Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0984250Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.0984495Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9207
2022-11-23T01:48:21.0984751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9208
2022-11-23T01:48:21.0985161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0985376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0985799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0986029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0986430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0986641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0987062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0987271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0987557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.0987838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.0988277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0988716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.0988986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.0989252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.0989679Z STAGE:2022-11-23 01:27:59 9208:9208 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0990024Z STAGE:2022-11-23 01:27:59 9207:9207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0990399Z STAGE:2022-11-23 01:27:59 9207:9207 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0990766Z STAGE:2022-11-23 01:27:59 9208:9208 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0991147Z STAGE:2022-11-23 01:27:59 9207:9207 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0991523Z STAGE:2022-11-23 01:27:59 9208:9208 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0992082Z STAGE:2022-11-23 01:27:59 9208:9208 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:27:59 9207:9207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0992106Z 
2022-11-23T01:48:21.0992470Z STAGE:2022-11-23 01:28:00 9207:9207 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0992831Z STAGE:2022-11-23 01:28:00 9208:9208 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0993429Z STAGE:2022-11-23 01:28:00 9208:9208 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:28:00 9207:9207 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0993450Z 
2022-11-23T01:48:21.0994052Z STAGE:2022-11-23 01:28:00 9208:9208 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:28:00 9207:9207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.0994075Z 
2022-11-23T01:48:21.0994453Z STAGE:2022-11-23 01:28:00 9207:9207 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0994817Z STAGE:2022-11-23 01:28:00 9208:9208 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.0995178Z STAGE:2022-11-23 01:28:00 9207:9207 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0995556Z STAGE:2022-11-23 01:28:00 9208:9208 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.0995701Z ok (6.117s)
2022-11-23T01:48:21.0995721Z 
2022-11-23T01:48:21.0996023Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0996174Z Ran 1 test in 6.117s
2022-11-23T01:48:21.0996194Z 
2022-11-23T01:48:21.0996321Z OK
2022-11-23T01:48:21.0996340Z 
2022-11-23T01:48:21.0996506Z Generating XML reports...
2022-11-23T01:48:21.0996996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012754.xml
2022-11-23T01:48:21.0997390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.0997612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.0998037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.0998268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.0998287Z 
2022-11-23T01:48:21.0998434Z Running tests...
2022-11-23T01:48:21.0998734Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0999087Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.0999375Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:48:21.0999395Z 
2022-11-23T01:48:21.0999695Z ----------------------------------------------------------------------
2022-11-23T01:48:21.0999827Z Ran 1 test in 0.002s
2022-11-23T01:48:21.0999847Z 
2022-11-23T01:48:21.0999992Z OK (skipped=1)
2022-11-23T01:48:21.1000061Z 
2022-11-23T01:48:21.1000223Z Generating XML reports...
2022-11-23T01:48:21.1000716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012803.xml
2022-11-23T01:48:21.1001127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1001343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1001761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1001997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1002018Z 
2022-11-23T01:48:21.1002170Z Running tests...
2022-11-23T01:48:21.1002456Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1002808Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1003108Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:48:21.1003129Z 
2022-11-23T01:48:21.1003424Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1003575Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1003595Z 
2022-11-23T01:48:21.1003742Z OK (skipped=1)
2022-11-23T01:48:21.1003761Z 
2022-11-23T01:48:21.1003925Z Generating XML reports...
2022-11-23T01:48:21.1004412Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012805.xml
2022-11-23T01:48:21.1004848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1005069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1005495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1005938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1005960Z 
2022-11-23T01:48:21.1006116Z Running tests...
2022-11-23T01:48:21.1006469Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1006828Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1007128Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T01:48:21.1007149Z 
2022-11-23T01:48:21.1007447Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1007585Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1007606Z 
2022-11-23T01:48:21.1007754Z OK (skipped=1)
2022-11-23T01:48:21.1007773Z 
2022-11-23T01:48:21.1007936Z Generating XML reports...
2022-11-23T01:48:21.1008421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012808.xml
2022-11-23T01:48:21.1008836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1009052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1009470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1009702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1009722Z 
2022-11-23T01:48:21.1009870Z Running tests...
2022-11-23T01:48:21.1010152Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1010507Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1010818Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T01:48:21.1010837Z 
2022-11-23T01:48:21.1011136Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1011370Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1011390Z 
2022-11-23T01:48:21.1011537Z OK (skipped=1)
2022-11-23T01:48:21.1011556Z 
2022-11-23T01:48:21.1011719Z Generating XML reports...
2022-11-23T01:48:21.1012210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012810.xml
2022-11-23T01:48:21.1012599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1012815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1013239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1013468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1013488Z 
2022-11-23T01:48:21.1013634Z Running tests...
2022-11-23T01:48:21.1013931Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1014283Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1014580Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:48:21.1014600Z 
2022-11-23T01:48:21.1014896Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1015029Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1015049Z 
2022-11-23T01:48:21.1015199Z OK (skipped=1)
2022-11-23T01:48:21.1015217Z 
2022-11-23T01:48:21.1015378Z Generating XML reports...
2022-11-23T01:48:21.1015922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012812.xml
2022-11-23T01:48:21.1016343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1016560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1016984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1017216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1017235Z 
2022-11-23T01:48:21.1017381Z Running tests...
2022-11-23T01:48:21.1017664Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1018016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1018333Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T01:48:21.1018353Z 
2022-11-23T01:48:21.1018649Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1018800Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1018819Z 
2022-11-23T01:48:21.1018966Z OK (skipped=1)
2022-11-23T01:48:21.1018985Z 
2022-11-23T01:48:21.1019150Z Generating XML reports...
2022-11-23T01:48:21.1019641Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012815.xml
2022-11-23T01:48:21.1020050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1020245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1020663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1020896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1020919Z 
2022-11-23T01:48:21.1021067Z Running tests...
2022-11-23T01:48:21.1021368Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1021718Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1022010Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:48:21.1022078Z 
2022-11-23T01:48:21.1022382Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1022516Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1022555Z 
2022-11-23T01:48:21.1022685Z OK (skipped=1)
2022-11-23T01:48:21.1022703Z 
2022-11-23T01:48:21.1022869Z Generating XML reports...
2022-11-23T01:48:21.1023359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012817.xml
2022-11-23T01:48:21.1023772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1023987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1024411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1024641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1024664Z 
2022-11-23T01:48:21.1024811Z Running tests...
2022-11-23T01:48:21.1025091Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1025443Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1025757Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:48:21.1025777Z 
2022-11-23T01:48:21.1026069Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1026219Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1026238Z 
2022-11-23T01:48:21.1026431Z OK (skipped=1)
2022-11-23T01:48:21.1026452Z 
2022-11-23T01:48:21.1026619Z Generating XML reports...
2022-11-23T01:48:21.1027113Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012820.xml
2022-11-23T01:48:21.1027525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1027725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1028146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1028376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1028396Z 
2022-11-23T01:48:21.1028544Z Running tests...
2022-11-23T01:48:21.1028843Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1029196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1029518Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:21.1029538Z 
2022-11-23T01:48:21.1029836Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1029995Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1030014Z 
2022-11-23T01:48:21.1030142Z OK (skipped=1)
2022-11-23T01:48:21.1030162Z 
2022-11-23T01:48:21.1030326Z Generating XML reports...
2022-11-23T01:48:21.1030808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012822.xml
2022-11-23T01:48:21.1031222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1031437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1031853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1032082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1032102Z 
2022-11-23T01:48:21.1032251Z Running tests...
2022-11-23T01:48:21.1032530Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1032935Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1033268Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:21.1033289Z 
2022-11-23T01:48:21.1033583Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1033735Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1033753Z 
2022-11-23T01:48:21.1033900Z OK (skipped=1)
2022-11-23T01:48:21.1033919Z 
2022-11-23T01:48:21.1034080Z Generating XML reports...
2022-11-23T01:48:21.1034567Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012824.xml
2022-11-23T01:48:21.1034977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1035173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1035598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1035827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1035847Z 
2022-11-23T01:48:21.1035995Z Running tests...
2022-11-23T01:48:21.1036295Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1036645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1037019Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:48:21.1037041Z 
2022-11-23T01:48:21.1037343Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1037494Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1037514Z 
2022-11-23T01:48:21.1037642Z OK (skipped=1)
2022-11-23T01:48:21.1037660Z 
2022-11-23T01:48:21.1037829Z Generating XML reports...
2022-11-23T01:48:21.1038316Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012827.xml
2022-11-23T01:48:21.1038726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1038942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1039359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1039589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1039613Z 
2022-11-23T01:48:21.1039761Z Running tests...
2022-11-23T01:48:21.1040063Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1040438Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1040784Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:48:21.1040808Z 
2022-11-23T01:48:21.1041108Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1041260Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1041280Z 
2022-11-23T01:48:21.1041426Z OK (skipped=1)
2022-11-23T01:48:21.1041445Z 
2022-11-23T01:48:21.1041606Z Generating XML reports...
2022-11-23T01:48:21.1042088Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012829.xml
2022-11-23T01:48:21.1042500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1042716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1043115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1043402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1043422Z 
2022-11-23T01:48:21.1043570Z Running tests...
2022-11-23T01:48:21.1043871Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1044221Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1044554Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:21.1044574Z 
2022-11-23T01:48:21.1044874Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1045031Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1045050Z 
2022-11-23T01:48:21.1045178Z OK (skipped=1)
2022-11-23T01:48:21.1045217Z 
2022-11-23T01:48:21.1045361Z Generating XML reports...
2022-11-23T01:48:21.1046076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012832.xml
2022-11-23T01:48:21.1046504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1046722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1047140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1047371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1047391Z 
2022-11-23T01:48:21.1047538Z Running tests...
2022-11-23T01:48:21.1047905Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1048251Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1048599Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:48:21.1048623Z 
2022-11-23T01:48:21.1048925Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1049077Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1049096Z 
2022-11-23T01:48:21.1049243Z OK (skipped=1)
2022-11-23T01:48:21.1049262Z 
2022-11-23T01:48:21.1049426Z Generating XML reports...
2022-11-23T01:48:21.1049910Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012834.xml
2022-11-23T01:48:21.1050317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1050537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1050936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1051167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1051187Z 
2022-11-23T01:48:21.1051332Z Running tests...
2022-11-23T01:48:21.1051636Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1051983Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1052317Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:21.1052337Z 
2022-11-23T01:48:21.1052636Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1052786Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1052806Z 
2022-11-23T01:48:21.1052951Z OK (skipped=1)
2022-11-23T01:48:21.1052970Z 
2022-11-23T01:48:21.1053119Z Generating XML reports...
2022-11-23T01:48:21.1053604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012836.xml
2022-11-23T01:48:21.1054016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1054300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1054721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1054955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1054975Z 
2022-11-23T01:48:21.1055123Z Running tests...
2022-11-23T01:48:21.1055423Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1055755Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1056102Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:48:21.1056122Z 
2022-11-23T01:48:21.1056424Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1056573Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1056596Z 
2022-11-23T01:48:21.1056745Z OK (skipped=1)
2022-11-23T01:48:21.1056765Z 
2022-11-23T01:48:21.1056927Z Generating XML reports...
2022-11-23T01:48:21.1057410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012839.xml
2022-11-23T01:48:21.1057816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1058033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1058433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1058726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1058747Z 
2022-11-23T01:48:21.1058900Z Running tests...
2022-11-23T01:48:21.1059207Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1059563Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1059894Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:21.1059915Z 
2022-11-23T01:48:21.1060211Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1060362Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1060382Z 
2022-11-23T01:48:21.1060531Z OK (skipped=1)
2022-11-23T01:48:21.1060550Z 
2022-11-23T01:48:21.1060694Z Generating XML reports...
2022-11-23T01:48:21.1061179Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012841.xml
2022-11-23T01:48:21.1061586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1061802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1062217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1062450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1062470Z 
2022-11-23T01:48:21.1062618Z Running tests...
2022-11-23T01:48:21.1062920Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1063271Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1063596Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:21.1063615Z 
2022-11-23T01:48:21.1063917Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1064068Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1064088Z 
2022-11-23T01:48:21.1064233Z OK (skipped=1)
2022-11-23T01:48:21.1064252Z 
2022-11-23T01:48:21.1064411Z Generating XML reports...
2022-11-23T01:48:21.1064898Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012844.xml
2022-11-23T01:48:21.1065367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1065584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1066006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1066218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1066238Z 
2022-11-23T01:48:21.1066388Z Running tests...
2022-11-23T01:48:21.1066692Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1067044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1067381Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:48:21.1067405Z 
2022-11-23T01:48:21.1067702Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1067853Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1067874Z 
2022-11-23T01:48:21.1068023Z OK (skipped=1)
2022-11-23T01:48:21.1068042Z 
2022-11-23T01:48:21.1068205Z Generating XML reports...
2022-11-23T01:48:21.1068672Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012846.xml
2022-11-23T01:48:21.1069170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1069391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1069812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1070043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1070066Z 
2022-11-23T01:48:21.1070216Z Running tests...
2022-11-23T01:48:21.1070516Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1070865Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1071193Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:48:21.1071233Z 
2022-11-23T01:48:21.1071510Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1071659Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1071678Z 
2022-11-23T01:48:21.1071828Z OK (skipped=1)
2022-11-23T01:48:21.1071847Z 
2022-11-23T01:48:21.1072008Z Generating XML reports...
2022-11-23T01:48:21.1072491Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012848.xml
2022-11-23T01:48:21.1072897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1073119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1073536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1073746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1073783Z 
2022-11-23T01:48:21.1073914Z Running tests...
2022-11-23T01:48:21.1074214Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1074568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1074906Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:21.1074926Z 
2022-11-23T01:48:21.1075223Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1075435Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1075454Z 
2022-11-23T01:48:21.1075601Z OK (skipped=1)
2022-11-23T01:48:21.1075620Z 
2022-11-23T01:48:21.1075784Z Generating XML reports...
2022-11-23T01:48:21.1076253Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012851.xml
2022-11-23T01:48:21.1076662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1076875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1077298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1077530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1077549Z 
2022-11-23T01:48:21.1077698Z Running tests...
2022-11-23T01:48:21.1078000Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1078360Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1078709Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:48:21.1078730Z 
2022-11-23T01:48:21.1079009Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1079160Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1079180Z 
2022-11-23T01:48:21.1079326Z OK (skipped=1)
2022-11-23T01:48:21.1079345Z 
2022-11-23T01:48:21.1079507Z Generating XML reports...
2022-11-23T01:48:21.1080038Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012853.xml
2022-11-23T01:48:21.1080452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1080669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1081090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1081321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1081341Z 
2022-11-23T01:48:21.1081471Z Running tests...
2022-11-23T01:48:21.1081771Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1082122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1082460Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:48:21.1082480Z 
2022-11-23T01:48:21.1082774Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1082926Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1082946Z 
2022-11-23T01:48:21.1083092Z OK (skipped=1)
2022-11-23T01:48:21.1083115Z 
2022-11-23T01:48:21.1083281Z Generating XML reports...
2022-11-23T01:48:21.1083750Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012856.xml
2022-11-23T01:48:21.1084164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1084377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1084796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1085030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1085050Z 
2022-11-23T01:48:21.1085198Z Running tests...
2022-11-23T01:48:21.1085495Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1086068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1086502Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:48:21.1086525Z 
2022-11-23T01:48:21.1086810Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1086965Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1086984Z 
2022-11-23T01:48:21.1087134Z OK (skipped=1)
2022-11-23T01:48:21.1087152Z 
2022-11-23T01:48:21.1087316Z Generating XML reports...
2022-11-23T01:48:21.1087804Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012858.xml
2022-11-23T01:48:21.1088216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1088432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1088853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1089087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1089108Z 
2022-11-23T01:48:21.1089236Z Running tests...
2022-11-23T01:48:21.1089540Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1089891Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1090191Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1090454Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10115
2022-11-23T01:48:21.1090777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10116
2022-11-23T01:48:21.1091201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1091420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1091825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1092060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1092464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1092678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1093090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1093321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1093607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1093893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1094334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1094758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1095032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1095300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1095580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.1095865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.1096307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1096738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1096940Z ok (5.671s)
2022-11-23T01:48:21.1096961Z 
2022-11-23T01:48:21.1097269Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1097401Z Ran 1 test in 5.671s
2022-11-23T01:48:21.1097420Z 
2022-11-23T01:48:21.1097557Z OK
2022-11-23T01:48:21.1097576Z 
2022-11-23T01:48:21.1097740Z Generating XML reports...
2022-11-23T01:48:21.1098230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012900.xml
2022-11-23T01:48:21.1098641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1098857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1099276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1099508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1099531Z 
2022-11-23T01:48:21.1099684Z Running tests...
2022-11-23T01:48:21.1099966Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1100318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1100617Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1100873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10236
2022-11-23T01:48:21.1101130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10237
2022-11-23T01:48:21.1101586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1101808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1102234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1102451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1102857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1103072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1103484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1103714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1104000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1104286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1104724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1105168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1105419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1105690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1105879Z skip: Need at least 3 CUDA devices (4.149s)
2022-11-23T01:48:21.1105899Z 
2022-11-23T01:48:21.1106203Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1106356Z Ran 1 test in 4.149s
2022-11-23T01:48:21.1106376Z 
2022-11-23T01:48:21.1106526Z OK (skipped=1)
2022-11-23T01:48:21.1106546Z 
2022-11-23T01:48:21.1106710Z Generating XML reports...
2022-11-23T01:48:21.1107201Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012909.xml
2022-11-23T01:48:21.1107589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1107857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1108275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1108504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1108524Z 
2022-11-23T01:48:21.1108672Z Running tests...
2022-11-23T01:48:21.1108971Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1109327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1109617Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-11-23T01:48:21.1109637Z 
2022-11-23T01:48:21.1109935Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1110071Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1110090Z 
2022-11-23T01:48:21.1110240Z OK (skipped=1)
2022-11-23T01:48:21.1110260Z 
2022-11-23T01:48:21.1110421Z Generating XML reports...
2022-11-23T01:48:21.1110907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012915.xml
2022-11-23T01:48:21.1111316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1111529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1111991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1112229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1112249Z 
2022-11-23T01:48:21.1112397Z Running tests...
2022-11-23T01:48:21.1112680Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1113037Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1113318Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1113577Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10378
2022-11-23T01:48:21.1113833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10379
2022-11-23T01:48:21.1114242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1114454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1114874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1115087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1115491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1115711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1116121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1116347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1116631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1116915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1117357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1117794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1118043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1118363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1118508Z ok (5.043s)
2022-11-23T01:48:21.1118528Z 
2022-11-23T01:48:21.1118836Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1118991Z Ran 1 test in 5.043s
2022-11-23T01:48:21.1119011Z 
2022-11-23T01:48:21.1119143Z OK
2022-11-23T01:48:21.1119162Z 
2022-11-23T01:48:21.1119324Z Generating XML reports...
2022-11-23T01:48:21.1119818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012918.xml
2022-11-23T01:48:21.1120211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1120431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1120849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1121084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1121103Z 
2022-11-23T01:48:21.1121252Z Running tests...
2022-11-23T01:48:21.1121554Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1121907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1122198Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1122503Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10487
2022-11-23T01:48:21.1122749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10488
2022-11-23T01:48:21.1123164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1123382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1123802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1124029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1124429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1124640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1125049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1125264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1125546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1126050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1126509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1126946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1127219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1127485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1127630Z ok (5.873s)
2022-11-23T01:48:21.1127650Z 
2022-11-23T01:48:21.1127958Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1128093Z Ran 1 test in 5.873s
2022-11-23T01:48:21.1128113Z 
2022-11-23T01:48:21.1128242Z OK
2022-11-23T01:48:21.1128262Z 
2022-11-23T01:48:21.1128425Z Generating XML reports...
2022-11-23T01:48:21.1128912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012925.xml
2022-11-23T01:48:21.1129405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1129623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1130044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1130274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1130295Z 
2022-11-23T01:48:21.1130443Z Running tests...
2022-11-23T01:48:21.1130731Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1131088Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1131387Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1131647Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10598
2022-11-23T01:48:21.1131906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10599
2022-11-23T01:48:21.1132317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1132533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1132950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1133163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1133635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1133856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1134271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1134505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1134794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1135081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1135523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1135960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1136218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1136486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1136764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.1137046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.1137486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1137916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1138062Z ok (5.051s)
2022-11-23T01:48:21.1138083Z 
2022-11-23T01:48:21.1138384Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1138519Z Ran 1 test in 5.051s
2022-11-23T01:48:21.1138560Z 
2022-11-23T01:48:21.1138673Z OK
2022-11-23T01:48:21.1138692Z 
2022-11-23T01:48:21.1138856Z Generating XML reports...
2022-11-23T01:48:21.1139344Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012934.xml
2022-11-23T01:48:21.1139752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1140030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1140490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1140725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1140745Z 
2022-11-23T01:48:21.1140896Z Running tests...
2022-11-23T01:48:21.1141178Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1141534Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1141839Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1142100Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10713
2022-11-23T01:48:21.1142359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10714
2022-11-23T01:48:21.1142772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1142989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1143409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1143640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1144071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1144292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1144715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1144943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1145231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1145514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1145950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1146384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1146637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1146909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1147107Z skip: Skipped due to small world size. (4.259s)
2022-11-23T01:48:21.1147127Z 
2022-11-23T01:48:21.1147433Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1147591Z Ran 1 test in 4.259s
2022-11-23T01:48:21.1147611Z 
2022-11-23T01:48:21.1147757Z OK (skipped=1)
2022-11-23T01:48:21.1147776Z 
2022-11-23T01:48:21.1147939Z Generating XML reports...
2022-11-23T01:48:21.1148429Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012941.xml
2022-11-23T01:48:21.1148841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1149038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1149462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1149693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1149713Z 
2022-11-23T01:48:21.1149862Z Running tests...
2022-11-23T01:48:21.1150166Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1150572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1150864Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1151125Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10822
2022-11-23T01:48:21.1151361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10823
2022-11-23T01:48:21.1151775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1151994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1152414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1152644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1153053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1153273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1153686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1153914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1154181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1154465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1154947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1155389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1155656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1155928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1156129Z skip: Skipped due to small world size. (4.251s)
2022-11-23T01:48:21.1156149Z 
2022-11-23T01:48:21.1156451Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1156603Z Ran 1 test in 4.251s
2022-11-23T01:48:21.1156622Z 
2022-11-23T01:48:21.1156750Z OK (skipped=1)
2022-11-23T01:48:21.1156769Z 
2022-11-23T01:48:21.1156931Z Generating XML reports...
2022-11-23T01:48:21.1157426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012948.xml
2022-11-23T01:48:21.1157836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1158051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1158475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1158704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1158724Z 
2022-11-23T01:48:21.1158872Z Running tests...
2022-11-23T01:48:21.1159155Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1159505Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1159806Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1160069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10931
2022-11-23T01:48:21.1160326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10932
2022-11-23T01:48:21.1160736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1161003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1161432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1161663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1162050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1162263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1162680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1162907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1163189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1163471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1163915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1164347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1164617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1164866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1165109Z skip: Skipped due to small world size. (4.211s)
2022-11-23T01:48:21.1165131Z 
2022-11-23T01:48:21.1165444Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1165601Z Ran 1 test in 4.212s
2022-11-23T01:48:21.1165621Z 
2022-11-23T01:48:21.1165983Z OK (skipped=1)
2022-11-23T01:48:21.1166005Z 
2022-11-23T01:48:21.1166174Z Generating XML reports...
2022-11-23T01:48:21.1166676Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012955.xml
2022-11-23T01:48:21.1167090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1167287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1167708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1167935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1167956Z 
2022-11-23T01:48:21.1168105Z Running tests...
2022-11-23T01:48:21.1168409Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1168759Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1169070Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1169333Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11040
2022-11-23T01:48:21.1169590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11041
2022-11-23T01:48:21.1169982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1170198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1170618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1170851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1171251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1171462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1171960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1172189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1172454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1172738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1173178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1173618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1173887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1174153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1174433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.1174712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.1175150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1175561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1175708Z ok (5.467s)
2022-11-23T01:48:21.1175729Z 
2022-11-23T01:48:21.1176093Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1176254Z Ran 1 test in 5.467s
2022-11-23T01:48:21.1176274Z 
2022-11-23T01:48:21.1176408Z OK
2022-11-23T01:48:21.1176427Z 
2022-11-23T01:48:21.1176590Z Generating XML reports...
2022-11-23T01:48:21.1177084Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013002.xml
2022-11-23T01:48:21.1177501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1177720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1178117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1178351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1178370Z 
2022-11-23T01:48:21.1178519Z Running tests...
2022-11-23T01:48:21.1178821Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1179177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1179620Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2022-11-23T01:48:21.1179644Z 
2022-11-23T01:48:21.1179951Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1180104Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1180123Z 
2022-11-23T01:48:21.1180269Z OK (skipped=1)
2022-11-23T01:48:21.1180289Z 
2022-11-23T01:48:21.1180435Z Generating XML reports...
2022-11-23T01:48:21.1180925Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013010.xml
2022-11-23T01:48:21.1181338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1181553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1181970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1182203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1182271Z 
2022-11-23T01:48:21.1182426Z Running tests...
2022-11-23T01:48:21.1182737Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1183094Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1183380Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1183639Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11188
2022-11-23T01:48:21.1183900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11189
2022-11-23T01:48:21.1184312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1184527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1184946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1185180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1185584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1185777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1186188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1186413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1186743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1187031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1187475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1187918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1188192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1188458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1188637Z skip: Skipped due to small world size. (4.172s)
2022-11-23T01:48:21.1188657Z 
2022-11-23T01:48:21.1188965Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1189113Z Ran 1 test in 4.172s
2022-11-23T01:48:21.1189138Z 
2022-11-23T01:48:21.1189287Z OK (skipped=1)
2022-11-23T01:48:21.1189307Z 
2022-11-23T01:48:21.1189468Z Generating XML reports...
2022-11-23T01:48:21.1189961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013012.xml
2022-11-23T01:48:21.1190380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1190594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1191015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1191225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1191245Z 
2022-11-23T01:48:21.1191395Z Running tests...
2022-11-23T01:48:21.1191698Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1192055Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1192360Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1192618Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11297
2022-11-23T01:48:21.1192933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11298
2022-11-23T01:48:21.1193349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1193544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1193964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1194192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1194602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1194813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1195230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1195460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1195754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1196042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1196464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1196899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1197217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1197493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1197637Z ok (4.307s)
2022-11-23T01:48:21.1197657Z 
2022-11-23T01:48:21.1197966Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1198125Z Ran 1 test in 4.307s
2022-11-23T01:48:21.1198144Z 
2022-11-23T01:48:21.1198277Z OK
2022-11-23T01:48:21.1198296Z 
2022-11-23T01:48:21.1198441Z Generating XML reports...
2022-11-23T01:48:21.1198931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013019.xml
2022-11-23T01:48:21.1199339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1199553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1199973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1200203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1200223Z 
2022-11-23T01:48:21.1200373Z Running tests...
2022-11-23T01:48:21.1200674Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1201033Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1201327Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1201585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11406
2022-11-23T01:48:21.1201841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11407
2022-11-23T01:48:21.1202251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1202469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1202886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1203116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1203519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1203767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1204186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1204418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1204704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1204985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1205428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1206065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1206339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1206611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1206736Z ok (4.139s)
2022-11-23T01:48:21.1206757Z 
2022-11-23T01:48:21.1207065Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1207216Z Ran 1 test in 4.139s
2022-11-23T01:48:21.1207236Z 
2022-11-23T01:48:21.1207370Z OK
2022-11-23T01:48:21.1207390Z 
2022-11-23T01:48:21.1207557Z Generating XML reports...
2022-11-23T01:48:21.1208147Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013026.xml
2022-11-23T01:48:21.1208575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1208793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1209215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1209432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1209451Z 
2022-11-23T01:48:21.1209602Z Running tests...
2022-11-23T01:48:21.1209905Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1210262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1210576Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:48:21.1210596Z 
2022-11-23T01:48:21.1210894Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1211045Z Ran 1 test in 0.003s
2022-11-23T01:48:21.1211064Z 
2022-11-23T01:48:21.1211210Z OK (skipped=1)
2022-11-23T01:48:21.1211229Z 
2022-11-23T01:48:21.1211372Z Generating XML reports...
2022-11-23T01:48:21.1211862Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013032.xml
2022-11-23T01:48:21.1212279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1212496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1212911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1213141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1213160Z 
2022-11-23T01:48:21.1213310Z Running tests...
2022-11-23T01:48:21.1213611Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1213961Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1214242Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-11-23T01:48:21.1214324Z 
2022-11-23T01:48:21.1214630Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1214784Z Ran 1 test in 0.003s
2022-11-23T01:48:21.1214803Z 
2022-11-23T01:48:21.1214948Z OK (skipped=1)
2022-11-23T01:48:21.1214967Z 
2022-11-23T01:48:21.1215129Z Generating XML reports...
2022-11-23T01:48:21.1215619Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013035.xml
2022-11-23T01:48:21.1216028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1216246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1216668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1216880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1216904Z 
2022-11-23T01:48:21.1217051Z Running tests...
2022-11-23T01:48:21.1217352Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1217703Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1218014Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:48:21.1218035Z 
2022-11-23T01:48:21.1218330Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1218481Z Ran 1 test in 0.003s
2022-11-23T01:48:21.1218501Z 
2022-11-23T01:48:21.1218646Z OK (skipped=1)
2022-11-23T01:48:21.1218714Z 
2022-11-23T01:48:21.1218884Z Generating XML reports...
2022-11-23T01:48:21.1219359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013037.xml
2022-11-23T01:48:21.1219772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1219997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1220412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1220646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1220665Z 
2022-11-23T01:48:21.1220815Z Running tests...
2022-11-23T01:48:21.1221114Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1221467Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1221753Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:48:21.1221790Z 
2022-11-23T01:48:21.1222073Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1222224Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1222247Z 
2022-11-23T01:48:21.1222400Z OK (skipped=1)
2022-11-23T01:48:21.1222419Z 
2022-11-23T01:48:21.1222581Z Generating XML reports...
2022-11-23T01:48:21.1223065Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013040.xml
2022-11-23T01:48:21.1223472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1223686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1224104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1224319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1224339Z 
2022-11-23T01:48:21.1224488Z Running tests...
2022-11-23T01:48:21.1224791Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1225143Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1225501Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:48:21.1225521Z 
2022-11-23T01:48:21.1225823Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1225975Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1225995Z 
2022-11-23T01:48:21.1226144Z OK (skipped=1)
2022-11-23T01:48:21.1226163Z 
2022-11-23T01:48:21.1226328Z Generating XML reports...
2022-11-23T01:48:21.1226800Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013042.xml
2022-11-23T01:48:21.1227210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1227424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1227839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1228073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1228092Z 
2022-11-23T01:48:21.1228238Z Running tests...
2022-11-23T01:48:21.1228537Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1228888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1229186Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:48:21.1229224Z 
2022-11-23T01:48:21.1229549Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1229707Z Ran 1 test in 0.003s
2022-11-23T01:48:21.1229726Z 
2022-11-23T01:48:21.1229875Z OK (skipped=1)
2022-11-23T01:48:21.1229895Z 
2022-11-23T01:48:21.1230058Z Generating XML reports...
2022-11-23T01:48:21.1230547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013044.xml
2022-11-23T01:48:21.1230963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1231179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1231598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1231810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1231849Z 
2022-11-23T01:48:21.1231978Z Running tests...
2022-11-23T01:48:21.1232286Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1232639Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1232937Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:48:21.1232961Z 
2022-11-23T01:48:21.1233262Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1233413Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1233433Z 
2022-11-23T01:48:21.1233582Z OK (skipped=1)
2022-11-23T01:48:21.1233601Z 
2022-11-23T01:48:21.1233765Z Generating XML reports...
2022-11-23T01:48:21.1234232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013047.xml
2022-11-23T01:48:21.1234642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1234859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1235276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1235509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1235528Z 
2022-11-23T01:48:21.1235763Z Running tests...
2022-11-23T01:48:21.1236066Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1236419Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1236702Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:48:21.1236742Z 
2022-11-23T01:48:21.1237022Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1237172Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1237192Z 
2022-11-23T01:48:21.1237338Z OK (skipped=1)
2022-11-23T01:48:21.1237357Z 
2022-11-23T01:48:21.1237521Z Generating XML reports...
2022-11-23T01:48:21.1238000Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013049.xml
2022-11-23T01:48:21.1238407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1238624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1239039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1239251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1239289Z 
2022-11-23T01:48:21.1239419Z Running tests...
2022-11-23T01:48:21.1239719Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1240069Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1240442Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1240711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11779
2022-11-23T01:48:21.1240969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11780
2022-11-23T01:48:21.1241378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1241597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1241995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1242224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1242623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1242835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1243247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1243474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1243758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1244045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1244468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1244906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1245176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1245448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1246044Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1246419Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1246796Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1247253Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1247642Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1248009Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1248378Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1248744Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1249114Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1249479Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1249868Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1250247Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1250609Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1250965Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1251317Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1251762Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1252143Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1252527Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1252895Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1253255Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1253625Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1253989Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1254370Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1254736Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1255102Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1255465Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1255837Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1256217Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1256587Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1256969Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1257336Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1257699Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1258051Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1258417Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1258852Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1259235Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1259598Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1259956Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1260328Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1260709Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1261075Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1261435Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1261805Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1262163Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1262533Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1262899Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1263331Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1263722Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1264085Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1264448Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1264798Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1265177Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1265545Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1265928Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1266291Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1266656Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1267025Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1267393Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1267772Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1268131Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1268496Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1268854Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1269227Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1269777Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1270291Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1270754Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1271118Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1271476Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1271825Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1272192Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1272577Z STAGE:2022-11-23 01:30:55 11780:11780 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1272957Z STAGE:2022-11-23 01:30:55 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1273101Z ok (4.213s)
2022-11-23T01:48:21.1273126Z 
2022-11-23T01:48:21.1273429Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1273581Z Ran 1 test in 4.213s
2022-11-23T01:48:21.1273601Z 
2022-11-23T01:48:21.1273731Z OK
2022-11-23T01:48:21.1273750Z 
2022-11-23T01:48:21.1273913Z Generating XML reports...
2022-11-23T01:48:21.1274388Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013052.xml
2022-11-23T01:48:21.1274801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1275064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1275494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1275728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1275747Z 
2022-11-23T01:48:21.1275901Z Running tests...
2022-11-23T01:48:21.1276205Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1276558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1276833Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1277632Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.628s)
2022-11-23T01:48:21.1277674Z 
2022-11-23T01:48:21.1277956Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1278107Z Ran 1 test in 1.628s
2022-11-23T01:48:21.1278127Z 
2022-11-23T01:48:21.1278277Z OK (skipped=1)
2022-11-23T01:48:21.1278296Z 
2022-11-23T01:48:21.1278463Z Generating XML reports...
2022-11-23T01:48:21.1278955Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013058.xml
2022-11-23T01:48:21.1279372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1279593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1280012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1280224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1280248Z 
2022-11-23T01:48:21.1280397Z Running tests...
2022-11-23T01:48:21.1280699Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1281054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1281361Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1281708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11926
2022-11-23T01:48:21.1281968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11927
2022-11-23T01:48:21.1282385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1282584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1283003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1283242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1283652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1283865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1284283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1284515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1284802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1285088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1285512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1286225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1286509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1286781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1287064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.1287346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.1287795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1288231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1288612Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1288954Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1289542Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1289568Z 
2022-11-23T01:48:21.1290178Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1290199Z 
2022-11-23T01:48:21.1290565Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1290929Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1291305Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1291691Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1292063Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1292514Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1292878Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1293240Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1293596Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1293968Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1294356Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1294741Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1295106Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1295476Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1295844Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1296225Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1296576Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1296957Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1297366Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1297732Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1298105Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1298471Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1298852Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1299232Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1299593Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1299940Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1300312Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1300910Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1300934Z 
2022-11-23T01:48:21.1301315Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1301674Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1302030Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1302401Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1302766Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1303150Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1303533Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1303935Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1304300Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1304670Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1305267Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1305287Z 
2022-11-23T01:48:21.1305671Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1306033Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1306442Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1306823Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1307193Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1307574Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1307939Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1308302Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1308712Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1309091Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1309682Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1309706Z 
2022-11-23T01:48:21.1310087Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1310447Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1310804Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1311174Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1311543Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1311906Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1312288Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1312654Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1313015Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1313384Z STAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1313972Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:31:06 11927:11927 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1313995Z 
2022-11-23T01:48:21.1314372Z STAGE:2022-11-23 01:31:06 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1314515Z ok (4.225s)
2022-11-23T01:48:21.1314535Z 
2022-11-23T01:48:21.1314836Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1315022Z Ran 1 test in 4.225s
2022-11-23T01:48:21.1315061Z 
2022-11-23T01:48:21.1315172Z OK
2022-11-23T01:48:21.1315190Z 
2022-11-23T01:48:21.1315357Z Generating XML reports...
2022-11-23T01:48:21.1315857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013103.xml
2022-11-23T01:48:21.1316271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1316486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1316912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1317143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1317164Z 
2022-11-23T01:48:21.1317312Z Running tests...
2022-11-23T01:48:21.1317597Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1317955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1318249Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1318506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12045
2022-11-23T01:48:21.1318763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12046
2022-11-23T01:48:21.1319174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1319435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1319864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1320076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1320480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1320696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1321119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1321348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1321632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1321916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1322358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1322796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1323048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1323317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1323515Z skip: Skipped due to small world size. (4.151s)
2022-11-23T01:48:21.1323535Z 
2022-11-23T01:48:21.1323836Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1323988Z Ran 1 test in 4.152s
2022-11-23T01:48:21.1324007Z 
2022-11-23T01:48:21.1324154Z OK (skipped=1)
2022-11-23T01:48:21.1324173Z 
2022-11-23T01:48:21.1324335Z Generating XML reports...
2022-11-23T01:48:21.1324830Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013109.xml
2022-11-23T01:48:21.1325243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1325439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1326125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1326366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1326387Z 
2022-11-23T01:48:21.1326537Z Running tests...
2022-11-23T01:48:21.1326844Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1327200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1327499Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1327760Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12154
2022-11-23T01:48:21.1327999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12155
2022-11-23T01:48:21.1328411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1328634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1329053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1329283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1329687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1329902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1330387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1330626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1330891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1331173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1331617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1332053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1332320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1332584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1333407Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:48:21.1333564Z   warnings.warn(
2022-11-23T01:48:21.1334371Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:48:21.1334526Z   warnings.warn(
2022-11-23T01:48:21.1334649Z ok (5.062s)
2022-11-23T01:48:21.1334669Z 
2022-11-23T01:48:21.1334971Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1335121Z Ran 1 test in 5.063s
2022-11-23T01:48:21.1335140Z 
2022-11-23T01:48:21.1335270Z OK
2022-11-23T01:48:21.1335289Z 
2022-11-23T01:48:21.1335454Z Generating XML reports...
2022-11-23T01:48:21.1335946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013116.xml
2022-11-23T01:48:21.1336354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1336637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1337040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1337274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1337293Z 
2022-11-23T01:48:21.1337442Z Running tests...
2022-11-23T01:48:21.1337744Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1338097Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1338404Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1339193Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.636s)
2022-11-23T01:48:21.1339217Z 
2022-11-23T01:48:21.1339519Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1339675Z Ran 1 test in 1.637s
2022-11-23T01:48:21.1339694Z 
2022-11-23T01:48:21.1339843Z OK (skipped=1)
2022-11-23T01:48:21.1339862Z 
2022-11-23T01:48:21.1340006Z Generating XML reports...
2022-11-23T01:48:21.1340534Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013124.xml
2022-11-23T01:48:21.1340993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1341215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1341639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1341869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1341892Z 
2022-11-23T01:48:21.1342041Z Running tests...
2022-11-23T01:48:21.1342341Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1342677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1343031Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1343826Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.638s)
2022-11-23T01:48:21.1343847Z 
2022-11-23T01:48:21.1344148Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1344301Z Ran 1 test in 1.638s
2022-11-23T01:48:21.1344324Z 
2022-11-23T01:48:21.1344474Z OK (skipped=1)
2022-11-23T01:48:21.1344493Z 
2022-11-23T01:48:21.1344657Z Generating XML reports...
2022-11-23T01:48:21.1345147Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013128.xml
2022-11-23T01:48:21.1345559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1345776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1346178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1346411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1346431Z 
2022-11-23T01:48:21.1346582Z Running tests...
2022-11-23T01:48:21.1346886Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1347240Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1347648Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1348438Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.597s)
2022-11-23T01:48:21.1348459Z 
2022-11-23T01:48:21.1348763Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1348916Z Ran 1 test in 1.597s
2022-11-23T01:48:21.1348936Z 
2022-11-23T01:48:21.1349083Z OK (skipped=1)
2022-11-23T01:48:21.1349102Z 
2022-11-23T01:48:21.1349246Z Generating XML reports...
2022-11-23T01:48:21.1349732Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013132.xml
2022-11-23T01:48:21.1350145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1350359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1350777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1351007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1351027Z 
2022-11-23T01:48:21.1351175Z Running tests...
2022-11-23T01:48:21.1351518Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1351864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1352169Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1352431Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12367
2022-11-23T01:48:21.1352694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12368
2022-11-23T01:48:21.1353105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1353322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1353740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1353970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1354376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1354572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1354981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1355214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1355496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1355781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1356220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1356656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1356928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1357197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1357476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ylegomj
2022-11-23T01:48:21.1357847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ylegomj/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1358142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkykl_z_1
2022-11-23T01:48:21.1358451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkykl_z_1/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1358727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1359004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1359146Z ok (5.511s)
2022-11-23T01:48:21.1359166Z 
2022-11-23T01:48:21.1359481Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1359614Z Ran 1 test in 5.512s
2022-11-23T01:48:21.1359634Z 
2022-11-23T01:48:21.1359769Z OK
2022-11-23T01:48:21.1359788Z 
2022-11-23T01:48:21.1359949Z Generating XML reports...
2022-11-23T01:48:21.1360438Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013136.xml
2022-11-23T01:48:21.1360853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1361071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1361491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1361722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1361742Z 
2022-11-23T01:48:21.1361892Z Running tests...
2022-11-23T01:48:21.1362230Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1362598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1362913Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1363175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12482
2022-11-23T01:48:21.1363434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12483
2022-11-23T01:48:21.1363844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1364058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1364479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1364695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1365106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1365320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1365950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1366193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1366481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1366768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1367218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1367663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1367913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1368180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1368567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8t3lpa_w
2022-11-23T01:48:21.1368878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8t3lpa_w/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1369171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfz_02qwj
2022-11-23T01:48:21.1369478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfz_02qwj/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1369755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1370034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1370304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1370559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1370701Z ok (5.550s)
2022-11-23T01:48:21.1370722Z 
2022-11-23T01:48:21.1371039Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1371193Z Ran 1 test in 5.551s
2022-11-23T01:48:21.1371213Z 
2022-11-23T01:48:21.1371346Z OK
2022-11-23T01:48:21.1371365Z 
2022-11-23T01:48:21.1371528Z Generating XML reports...
2022-11-23T01:48:21.1372022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013144.xml
2022-11-23T01:48:21.1372432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1372627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1373110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1373352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1373372Z 
2022-11-23T01:48:21.1373524Z Running tests...
2022-11-23T01:48:21.1373834Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1374186Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1374498Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1375293Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.636s)
2022-11-23T01:48:21.1375317Z 
2022-11-23T01:48:21.1375621Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1375754Z Ran 1 test in 1.637s
2022-11-23T01:48:21.1375795Z 
2022-11-23T01:48:21.1375925Z OK (skipped=1)
2022-11-23T01:48:21.1375944Z 
2022-11-23T01:48:21.1376111Z Generating XML reports...
2022-11-23T01:48:21.1376606Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013152.xml
2022-11-23T01:48:21.1377017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1377233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1377654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1377888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1377908Z 
2022-11-23T01:48:21.1378060Z Running tests...
2022-11-23T01:48:21.1378346Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1378698Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1379027Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1379872Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.601s)
2022-11-23T01:48:21.1379893Z 
2022-11-23T01:48:21.1380194Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1380351Z Ran 1 test in 1.601s
2022-11-23T01:48:21.1380370Z 
2022-11-23T01:48:21.1380520Z OK (skipped=1)
2022-11-23T01:48:21.1380539Z 
2022-11-23T01:48:21.1380710Z Generating XML reports...
2022-11-23T01:48:21.1381199Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013156.xml
2022-11-23T01:48:21.1381611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1381812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1382235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1382466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1382485Z 
2022-11-23T01:48:21.1382635Z Running tests...
2022-11-23T01:48:21.1382936Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1383285Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1383664Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1383930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12665
2022-11-23T01:48:21.1384168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12666
2022-11-23T01:48:21.1384583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1384798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1385218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1385448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1385853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1386068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1386482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1386710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1386977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1387264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1387704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1388139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1388406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1388676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1388974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryzxywyq
2022-11-23T01:48:21.1389283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryzxywyq/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1389630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54fgo7ob
2022-11-23T01:48:21.1389919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54fgo7ob/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1390177Z <class 'torch.nn.parameter.Parameter'>
2022-11-23T01:48:21.1390322Z ok (5.144s)
2022-11-23T01:48:21.1390343Z 
2022-11-23T01:48:21.1390645Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1390799Z Ran 1 test in 5.144s
2022-11-23T01:48:21.1390819Z 
2022-11-23T01:48:21.1390951Z OK
2022-11-23T01:48:21.1390971Z 
2022-11-23T01:48:21.1391135Z Generating XML reports...
2022-11-23T01:48:21.1391633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013200.xml
2022-11-23T01:48:21.1392025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1392242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1392665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1392896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1392916Z 
2022-11-23T01:48:21.1393065Z Running tests...
2022-11-23T01:48:21.1393367Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1393719Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1394110Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1394381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12776
2022-11-23T01:48:21.1394621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12777
2022-11-23T01:48:21.1395034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1395257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1395676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1395906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1396309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1396523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1396937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1397147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1397437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1397724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1398164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1398597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1398867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1399137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1399437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivcmkesn
2022-11-23T01:48:21.1399747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivcmkesn/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1400023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8433tjfi
2022-11-23T01:48:21.1400384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8433tjfi/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1400530Z ok (5.054s)
2022-11-23T01:48:21.1400550Z 
2022-11-23T01:48:21.1400858Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1401015Z Ran 1 test in 5.054s
2022-11-23T01:48:21.1401035Z 
2022-11-23T01:48:21.1401168Z OK
2022-11-23T01:48:21.1401187Z 
2022-11-23T01:48:21.1401350Z Generating XML reports...
2022-11-23T01:48:21.1401845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013208.xml
2022-11-23T01:48:21.1402240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1402457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1402875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1403110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1403129Z 
2022-11-23T01:48:21.1403278Z Running tests...
2022-11-23T01:48:21.1403581Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1403933Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1404235Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1404569Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12887
2022-11-23T01:48:21.1404812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12888
2022-11-23T01:48:21.1405222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1405439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1406081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1406320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1406728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1406944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1407356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1407571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1407858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1408142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1408587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1409022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1409293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1409560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1409857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15j7n5dg
2022-11-23T01:48:21.1410167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15j7n5dg/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1410442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fksjm3_
2022-11-23T01:48:21.1410747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fksjm3_/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1411104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1411380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1411524Z ok (5.522s)
2022-11-23T01:48:21.1411545Z 
2022-11-23T01:48:21.1411852Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1412003Z Ran 1 test in 5.522s
2022-11-23T01:48:21.1412023Z 
2022-11-23T01:48:21.1412155Z OK
2022-11-23T01:48:21.1412175Z 
2022-11-23T01:48:21.1412341Z Generating XML reports...
2022-11-23T01:48:21.1412818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013216.xml
2022-11-23T01:48:21.1413231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1413447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1413874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1414107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1414126Z 
2022-11-23T01:48:21.1414276Z Running tests...
2022-11-23T01:48:21.1414578Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1414932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1415301Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1415568Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13002
2022-11-23T01:48:21.1415828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13003
2022-11-23T01:48:21.1416247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1416469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1416888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1417116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1417520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1417739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1418136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1418366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1418647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1418936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1419379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1419811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1420078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1420341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1420644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb49tjfqq
2022-11-23T01:48:21.1420938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb49tjfqq/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1421231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpry2tugvz
2022-11-23T01:48:21.1421597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpry2tugvz/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1422433Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:21.1422581Z ok (5.508s)
2022-11-23T01:48:21.1422601Z 
2022-11-23T01:48:21.1422908Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1423060Z Ran 1 test in 5.508s
2022-11-23T01:48:21.1423080Z 
2022-11-23T01:48:21.1423210Z OK
2022-11-23T01:48:21.1423229Z 
2022-11-23T01:48:21.1423398Z Generating XML reports...
2022-11-23T01:48:21.1423893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013224.xml
2022-11-23T01:48:21.1424286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1424501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1424923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1425202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1425224Z 
2022-11-23T01:48:21.1425375Z Running tests...
2022-11-23T01:48:21.1425683Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1426037Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1426356Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1427150Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.620s)
2022-11-23T01:48:21.1427171Z 
2022-11-23T01:48:21.1427472Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1427604Z Ran 1 test in 1.620s
2022-11-23T01:48:21.1427623Z 
2022-11-23T01:48:21.1427776Z OK (skipped=1)
2022-11-23T01:48:21.1427796Z 
2022-11-23T01:48:21.1427960Z Generating XML reports...
2022-11-23T01:48:21.1428449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013232.xml
2022-11-23T01:48:21.1428856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1429075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1429496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1429725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1429746Z 
2022-11-23T01:48:21.1429877Z Running tests...
2022-11-23T01:48:21.1430179Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1430536Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1430835Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1431093Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13151
2022-11-23T01:48:21.1431350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13152
2022-11-23T01:48:21.1431819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1432035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1432454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1432665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1433073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1433288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1433703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1433929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1434219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1434505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1434945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1435361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1435630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1435983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfn8ilyw9
2022-11-23T01:48:21.1436303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfn8ilyw9/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1436573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1436874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfo9jshbk
2022-11-23T01:48:21.1437184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfo9jshbk/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1438121Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:48:21.1439052Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:48:21.1440263Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T01:48:21.1440577Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-11-23T01:48:21.1442134Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T01:48:21.1442499Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-11-23T01:48:21.1442782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1443064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1443991Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:48:21.1444945Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:48:21.1446124Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:48:21.1447063Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:48:21.1447990Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:48:21.1448905Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:48:21.1449819Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:48:21.1450736Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:48:21.1451643Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:48:21.1452634Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:48:21.1452782Z ok (4.355s)
2022-11-23T01:48:21.1452805Z 
2022-11-23T01:48:21.1453093Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1453254Z Ran 1 test in 4.355s
2022-11-23T01:48:21.1453274Z 
2022-11-23T01:48:21.1453408Z OK
2022-11-23T01:48:21.1453428Z 
2022-11-23T01:48:21.1453595Z Generating XML reports...
2022-11-23T01:48:21.1454087Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013236.xml
2022-11-23T01:48:21.1454505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1454723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1455148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1455380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1455401Z 
2022-11-23T01:48:21.1455530Z Running tests...
2022-11-23T01:48:21.1455833Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1456239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1456537Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1457326Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.647s)
2022-11-23T01:48:21.1457351Z 
2022-11-23T01:48:21.1457654Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1457805Z Ran 1 test in 1.647s
2022-11-23T01:48:21.1457824Z 
2022-11-23T01:48:21.1457971Z OK (skipped=1)
2022-11-23T01:48:21.1457991Z 
2022-11-23T01:48:21.1458157Z Generating XML reports...
2022-11-23T01:48:21.1458628Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013243.xml
2022-11-23T01:48:21.1459045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1459262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1459685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1459918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1459938Z 
2022-11-23T01:48:21.1460086Z Running tests...
2022-11-23T01:48:21.1460388Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1460742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1461051Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1461295Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13298
2022-11-23T01:48:21.1461555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13299
2022-11-23T01:48:21.1461968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1462183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1462660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1462892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1463296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1463513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1463907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1464143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1464432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1464714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1465161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1465599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1465869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1466137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1466435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppu4afs4a
2022-11-23T01:48:21.1466780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppu4afs4a/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1467080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfz9h_8gk
2022-11-23T01:48:21.1467387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfz9h_8gk/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1468232Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T01:48:21.1468606Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T01:48:21.1469432Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T01:48:21.1469803Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T01:48:21.1469957Z ok (5.551s)
2022-11-23T01:48:21.1469980Z 
2022-11-23T01:48:21.1470280Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1470430Z Ran 1 test in 5.551s
2022-11-23T01:48:21.1470450Z 
2022-11-23T01:48:21.1470576Z OK
2022-11-23T01:48:21.1470595Z 
2022-11-23T01:48:21.1470754Z Generating XML reports...
2022-11-23T01:48:21.1471231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013247.xml
2022-11-23T01:48:21.1471644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1471862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1472282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1472509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1472575Z 
2022-11-23T01:48:21.1472725Z Running tests...
2022-11-23T01:48:21.1473028Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1473380Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1473694Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1474470Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2022-11-23T01:48:21.1474508Z 
2022-11-23T01:48:21.1474789Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1474939Z Ran 1 test in 1.633s
2022-11-23T01:48:21.1474958Z 
2022-11-23T01:48:21.1475102Z OK (skipped=1)
2022-11-23T01:48:21.1475126Z 
2022-11-23T01:48:21.1475288Z Generating XML reports...
2022-11-23T01:48:21.1475772Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013255.xml
2022-11-23T01:48:21.1476177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1476390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1476806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1477082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1477104Z 
2022-11-23T01:48:21.1477240Z Running tests...
2022-11-23T01:48:21.1477542Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1477892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1478205Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1478986Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2022-11-23T01:48:21.1479007Z 
2022-11-23T01:48:21.1479303Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1479454Z Ran 1 test in 1.633s
2022-11-23T01:48:21.1479473Z 
2022-11-23T01:48:21.1479622Z OK (skipped=1)
2022-11-23T01:48:21.1479642Z 
2022-11-23T01:48:21.1479802Z Generating XML reports...
2022-11-23T01:48:21.1480274Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013259.xml
2022-11-23T01:48:21.1480681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1480901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1481321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1481548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1481567Z 
2022-11-23T01:48:21.1481712Z Running tests...
2022-11-23T01:48:21.1482009Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1482362Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1482692Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1482931Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13511
2022-11-23T01:48:21.1483188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13512
2022-11-23T01:48:21.1483653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1483862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1484276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1484502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1484909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1485123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1485515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1485962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1486264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1486544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1486986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1487420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1487689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1488027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1488313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.1488577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.1489019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1489448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1489742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdizsio4o
2022-11-23T01:48:21.1490048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdizsio4o/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1490345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstmjdy21
2022-11-23T01:48:21.1490653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstmjdy21/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1490927Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1491200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1491461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1491729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1491868Z ok (5.957s)
2022-11-23T01:48:21.1491888Z 
2022-11-23T01:48:21.1492190Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1492339Z Ran 1 test in 5.957s
2022-11-23T01:48:21.1492359Z 
2022-11-23T01:48:21.1492486Z OK
2022-11-23T01:48:21.1492505Z 
2022-11-23T01:48:21.1492666Z Generating XML reports...
2022-11-23T01:48:21.1493163Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013303.xml
2022-11-23T01:48:21.1493557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1493775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1494268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1494496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1494517Z 
2022-11-23T01:48:21.1494662Z Running tests...
2022-11-23T01:48:21.1494959Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1495310Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1495627Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1495885Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13632
2022-11-23T01:48:21.1496124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13633
2022-11-23T01:48:21.1496533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1496749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1497164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1497393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1497791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1498001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1498461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1498680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1498965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1499251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1499691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1500123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1500390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1500705Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:48:21.1500966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1501268Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:48:21.1501548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdjepzqah
2022-11-23T01:48:21.1501864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdjepzqah/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1502154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibck9ihr
2022-11-23T01:48:21.1502463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibck9ihr/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1502735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1503008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1503276Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1503546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1503863Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:48:21.1504211Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:48:21.1504523Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:48:21.1504827Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:48:21.1505095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1505365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1505633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1505901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1506211Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:48:21.1506527Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:48:21.1506824Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T01:48:21.1507131Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T01:48:21.1507399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1507726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1508002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1508267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1508407Z ok (6.344s)
2022-11-23T01:48:21.1508427Z 
2022-11-23T01:48:21.1508741Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1508875Z Ran 1 test in 6.344s
2022-11-23T01:48:21.1508911Z 
2022-11-23T01:48:21.1509022Z OK
2022-11-23T01:48:21.1509041Z 
2022-11-23T01:48:21.1509200Z Generating XML reports...
2022-11-23T01:48:21.1509695Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013312.xml
2022-11-23T01:48:21.1510100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1510311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1510733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1510960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1510980Z 
2022-11-23T01:48:21.1511126Z Running tests...
2022-11-23T01:48:21.1511414Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1511764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1512071Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1512861Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.646s)
2022-11-23T01:48:21.1512888Z 
2022-11-23T01:48:21.1513186Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1513335Z Ran 1 test in 1.646s
2022-11-23T01:48:21.1513355Z 
2022-11-23T01:48:21.1513498Z OK (skipped=1)
2022-11-23T01:48:21.1513517Z 
2022-11-23T01:48:21.1513678Z Generating XML reports...
2022-11-23T01:48:21.1514219Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013321.xml
2022-11-23T01:48:21.1514627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1514824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1515240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1515466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1515486Z 
2022-11-23T01:48:21.1515634Z Running tests...
2022-11-23T01:48:21.1515933Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1516282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1516591Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1516854Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13781
2022-11-23T01:48:21.1517092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13782
2022-11-23T01:48:21.1520004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1520221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1520639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1520921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1521327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1521537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1521951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1522177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1522444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1522726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1523167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1523605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1523872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1524458Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:48:21.1524728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1525303Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:48:21.1525597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mu4g_5a
2022-11-23T01:48:21.1526130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mu4g_5a/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1526428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_vyrc29p
2022-11-23T01:48:21.1526800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_vyrc29p/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1527077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1527349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1527661Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T01:48:21.1527977Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T01:48:21.1528316Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T01:48:21.1528652Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T01:48:21.1529025Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T01:48:21.1529392Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T01:48:21.1529753Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T01:48:21.1530150Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T01:48:21.1530434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1530704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1530845Z ok (5.649s)
2022-11-23T01:48:21.1530870Z 
2022-11-23T01:48:21.1531182Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1531333Z Ran 1 test in 5.649s
2022-11-23T01:48:21.1531352Z 
2022-11-23T01:48:21.1531480Z OK
2022-11-23T01:48:21.1531499Z 
2022-11-23T01:48:21.1531660Z Generating XML reports...
2022-11-23T01:48:21.1532134Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013325.xml
2022-11-23T01:48:21.1532549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1532767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1533188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1533417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1533437Z 
2022-11-23T01:48:21.1533589Z Running tests...
2022-11-23T01:48:21.1533888Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1534238Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1534590Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1534833Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13896
2022-11-23T01:48:21.1535087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13897
2022-11-23T01:48:21.1535500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1535714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1536128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1536410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1536814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1537027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1537443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1537654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1537941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1538223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1538662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1539100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1539367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1539632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1539929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc5mhvssz
2022-11-23T01:48:21.1540239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc5mhvssz/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1540605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphlmziru4
2022-11-23T01:48:21.1540920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphlmziru4/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1541192Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1541464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1541739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1542000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1542140Z ok (5.756s)
2022-11-23T01:48:21.1542160Z 
2022-11-23T01:48:21.1542467Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1542602Z Ran 1 test in 5.757s
2022-11-23T01:48:21.1542622Z 
2022-11-23T01:48:21.1542750Z OK
2022-11-23T01:48:21.1542769Z 
2022-11-23T01:48:21.1542931Z Generating XML reports...
2022-11-23T01:48:21.1543427Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013333.xml
2022-11-23T01:48:21.1543834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1544046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1544466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1544693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1544712Z 
2022-11-23T01:48:21.1544858Z Running tests...
2022-11-23T01:48:21.1545144Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1545492Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1545848Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1546103Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14041
2022-11-23T01:48:21.1546356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14042
2022-11-23T01:48:21.1546766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1547035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1547453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1547664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1548065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1548278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1548702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1548927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1549207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1549494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1549932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1550365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1550617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1550931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1551233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpihohg4fh
2022-11-23T01:48:21.1551544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpihohg4fh/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1551840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo70rpjwt
2022-11-23T01:48:21.1552144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo70rpjwt/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1552419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1552690Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1552947Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1553219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1553363Z ok (5.639s)
2022-11-23T01:48:21.1553384Z 
2022-11-23T01:48:21.1553692Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1553841Z Ran 1 test in 5.639s
2022-11-23T01:48:21.1553861Z 
2022-11-23T01:48:21.1553990Z OK
2022-11-23T01:48:21.1554009Z 
2022-11-23T01:48:21.1554170Z Generating XML reports...
2022-11-23T01:48:21.1554668Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013342.xml
2022-11-23T01:48:21.1555077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1555275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1555692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1555921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1555940Z 
2022-11-23T01:48:21.1556090Z Running tests...
2022-11-23T01:48:21.1556387Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1556739Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1557148Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1557462Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14186
2022-11-23T01:48:21.1557716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14187
2022-11-23T01:48:21.1558111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1558322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1558745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1558974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1559373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1559586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1559994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1560221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1560487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1560770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1561253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1561695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1561963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1562229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1562526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr90dwsst
2022-11-23T01:48:21.1562832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr90dwsst/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1563124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6eumjgw0
2022-11-23T01:48:21.1563412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6eumjgw0/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1563687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1563960Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1564230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1564502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1564648Z ok (5.822s)
2022-11-23T01:48:21.1564668Z 
2022-11-23T01:48:21.1564974Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1565124Z Ran 1 test in 5.822s
2022-11-23T01:48:21.1565145Z 
2022-11-23T01:48:21.1565256Z OK
2022-11-23T01:48:21.1565292Z 
2022-11-23T01:48:21.1565436Z Generating XML reports...
2022-11-23T01:48:21.1566142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013350.xml
2022-11-23T01:48:21.1566567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1566782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1567201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1567429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1567565Z 
2022-11-23T01:48:21.1567716Z Running tests...
2022-11-23T01:48:21.1568018Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1568354Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1568764Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1569023Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14331
2022-11-23T01:48:21.1569279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14332
2022-11-23T01:48:21.1569692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1569904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1570326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1570553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1570954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1571148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1571561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1571850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1572141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1572422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1572858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1573296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1573565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1573812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1574110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53tnnhvp
2022-11-23T01:48:21.1574422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53tnnhvp/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1574713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6_tvr9_
2022-11-23T01:48:21.1575011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6_tvr9_/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1575286Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1575558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1575827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1576097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1576222Z ok (5.759s)
2022-11-23T01:48:21.1576242Z 
2022-11-23T01:48:21.1576555Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1576706Z Ran 1 test in 5.759s
2022-11-23T01:48:21.1576730Z 
2022-11-23T01:48:21.1576860Z OK
2022-11-23T01:48:21.1576879Z 
2022-11-23T01:48:21.1577039Z Generating XML reports...
2022-11-23T01:48:21.1577528Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013358.xml
2022-11-23T01:48:21.1577935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1578203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1578606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1578836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1578856Z 
2022-11-23T01:48:21.1579001Z Running tests...
2022-11-23T01:48:21.1579298Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1579656Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1580065Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1580320Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14476
2022-11-23T01:48:21.1580579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14477
2022-11-23T01:48:21.1580993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1581189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1581603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1581831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1582291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1582509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1582916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1583146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1583429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1583710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1584134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1584569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1584836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1585101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1585394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp08wn37qv
2022-11-23T01:48:21.1585702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp08wn37qv/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1585995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyx0p4bjl
2022-11-23T01:48:21.1586297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyx0p4bjl/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1586572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1586826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1587101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1587373Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1587513Z ok (5.707s)
2022-11-23T01:48:21.1587533Z 
2022-11-23T01:48:21.1587838Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1588045Z Ran 1 test in 5.707s
2022-11-23T01:48:21.1588065Z 
2022-11-23T01:48:21.1588191Z OK
2022-11-23T01:48:21.1588210Z 
2022-11-23T01:48:21.1588371Z Generating XML reports...
2022-11-23T01:48:21.1588845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013407.xml
2022-11-23T01:48:21.1589254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1589469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1589891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1590121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1590141Z 
2022-11-23T01:48:21.1590287Z Running tests...
2022-11-23T01:48:21.1590583Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1590942Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1591348Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1591591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14621
2022-11-23T01:48:21.1591846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14622
2022-11-23T01:48:21.1592255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1592515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1592939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1593167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1593572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1593780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1594174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1594403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1594687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1594970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1595406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1595839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1596111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1596376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1596670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lbly1qj
2022-11-23T01:48:21.1596959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lbly1qj/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1597252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcqvb9p1l
2022-11-23T01:48:21.1597560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcqvb9p1l/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1597834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1598106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1598431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1598699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1598838Z ok (5.821s)
2022-11-23T01:48:21.1598859Z 
2022-11-23T01:48:21.1599164Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1599300Z Ran 1 test in 5.821s
2022-11-23T01:48:21.1599319Z 
2022-11-23T01:48:21.1599448Z OK
2022-11-23T01:48:21.1599467Z 
2022-11-23T01:48:21.1599626Z Generating XML reports...
2022-11-23T01:48:21.1600121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013415.xml
2022-11-23T01:48:21.1600529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1600741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1601157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1601392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1601412Z 
2022-11-23T01:48:21.1601543Z Running tests...
2022-11-23T01:48:21.1601846Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1602199Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1602651Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1602916Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14766
2022-11-23T01:48:21.1603171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14767
2022-11-23T01:48:21.1603582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1603798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1604216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1604430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1604832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1605043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1605456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1605884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1606175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1606500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1606948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1607380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1607633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1607898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1608193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4meerr3s
2022-11-23T01:48:21.1608499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4meerr3s/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1608790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabpfg0mz
2022-11-23T01:48:21.1609179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabpfg0mz/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1609452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1609722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1609976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1610248Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1610389Z ok (5.731s)
2022-11-23T01:48:21.1610409Z 
2022-11-23T01:48:21.1610720Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1610870Z Ran 1 test in 5.731s
2022-11-23T01:48:21.1610890Z 
2022-11-23T01:48:21.1611017Z OK
2022-11-23T01:48:21.1611036Z 
2022-11-23T01:48:21.1611194Z Generating XML reports...
2022-11-23T01:48:21.1611685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013423.xml
2022-11-23T01:48:21.1612098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1612296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1612714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1612940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1612960Z 
2022-11-23T01:48:21.1613107Z Running tests...
2022-11-23T01:48:21.1613465Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1613828Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1614233Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1614495Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14911
2022-11-23T01:48:21.1614731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14912
2022-11-23T01:48:21.1615143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1615357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1615776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1616005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1616402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1616613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1617027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1617252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1617520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1617804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1618243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1618743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1619052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1619355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1619884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgeafgqk
2022-11-23T01:48:21.1620236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgeafgqk/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1620515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwqivy1qv
2022-11-23T01:48:21.1620860Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwqivy1qv/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1621168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1621479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1621787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1622087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1622308Z ok (5.857s)
2022-11-23T01:48:21.1622329Z 
2022-11-23T01:48:21.1622720Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1622856Z Ran 1 test in 5.857s
2022-11-23T01:48:21.1622876Z 
2022-11-23T01:48:21.1623042Z OK
2022-11-23T01:48:21.1623062Z 
2022-11-23T01:48:21.1623261Z Generating XML reports...
2022-11-23T01:48:21.1623798Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013431.xml
2022-11-23T01:48:21.1624245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1624544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1625020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1625283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1625308Z 
2022-11-23T01:48:21.1625528Z Running tests...
2022-11-23T01:48:21.1625816Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1626204Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1626650Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1626946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15056
2022-11-23T01:48:21.1627240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15057
2022-11-23T01:48:21.1627699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1627991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1628446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1628663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1629145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1629395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1629844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1630106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1630441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1630758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1631234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1631762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1632014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1632350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1632682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnh34fmn5
2022-11-23T01:48:21.1633041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnh34fmn5/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1633371Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwv6abifp
2022-11-23T01:48:21.1633716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwv6abifp/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1634051Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1634368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1634677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1634929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1635142Z ok (5.763s)
2022-11-23T01:48:21.1635163Z 
2022-11-23T01:48:21.1635519Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1635705Z Ran 1 test in 5.764s
2022-11-23T01:48:21.1635726Z 
2022-11-23T01:48:21.1635888Z OK
2022-11-23T01:48:21.1635954Z 
2022-11-23T01:48:21.1636156Z Generating XML reports...
2022-11-23T01:48:21.1636688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013440.xml
2022-11-23T01:48:21.1637139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1637343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1637802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1638113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1638133Z 
2022-11-23T01:48:21.1638317Z Running tests...
2022-11-23T01:48:21.1638652Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1639041Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1639521Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1639818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15201
2022-11-23T01:48:21.1640112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15202
2022-11-23T01:48:21.1640549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1640818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1641310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1641578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1642022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1642269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1642720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1642983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1643318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1643640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1644121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1644633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1644941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1645243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1645600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3wtkq03
2022-11-23T01:48:21.1646183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3wtkq03/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1646526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wmtkbhm
2022-11-23T01:48:21.1646818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wmtkbhm/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1647128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1647437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1647791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1648170Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1648356Z ok (5.731s)
2022-11-23T01:48:21.1648376Z 
2022-11-23T01:48:21.1648739Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1648930Z Ran 1 test in 5.731s
2022-11-23T01:48:21.1648955Z 
2022-11-23T01:48:21.1649117Z OK
2022-11-23T01:48:21.1649138Z 
2022-11-23T01:48:21.1649284Z Generating XML reports...
2022-11-23T01:48:21.1649822Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013448.xml
2022-11-23T01:48:21.1650265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1650556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1651015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1651333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1651354Z 
2022-11-23T01:48:21.1651536Z Running tests...
2022-11-23T01:48:21.1651875Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1652214Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1652609Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1652904Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15346
2022-11-23T01:48:21.1653194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15347
2022-11-23T01:48:21.1653682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1653943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1654401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1654667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1655102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1655377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1655836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1656098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1656444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1656801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1657320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1657794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1658098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1658407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1658688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp921jdb73
2022-11-23T01:48:21.1659030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp921jdb73/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1659373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8qnk9re
2022-11-23T01:48:21.1659713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8qnk9re/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1660109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1660427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1660736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1661049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1661175Z ok (5.840s)
2022-11-23T01:48:21.1661196Z 
2022-11-23T01:48:21.1661546Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1661742Z Ran 1 test in 5.840s
2022-11-23T01:48:21.1661762Z 
2022-11-23T01:48:21.1661928Z OK
2022-11-23T01:48:21.1661947Z 
2022-11-23T01:48:21.1662143Z Generating XML reports...
2022-11-23T01:48:21.1662752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013456.xml
2022-11-23T01:48:21.1663206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1663455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1663911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1664247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1664267Z 
2022-11-23T01:48:21.1676008Z Running tests...
2022-11-23T01:48:21.1676374Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1676745Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1677099Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1677359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15491
2022-11-23T01:48:21.1677621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15492
2022-11-23T01:48:21.1678044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1678243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1678792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1679024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1679432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1679645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1680061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1680299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1680588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1680872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1681304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1681745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1682017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1682282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1682579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwulh817q
2022-11-23T01:48:21.1682941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwulh817q/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1683243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhh6h1jv
2022-11-23T01:48:21.1683551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhh6h1jv/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1683815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1684092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1684367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1684640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1684781Z ok (5.745s)
2022-11-23T01:48:21.1684803Z 
2022-11-23T01:48:21.1685110Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1685268Z Ran 1 test in 5.745s
2022-11-23T01:48:21.1685289Z 
2022-11-23T01:48:21.1685421Z OK
2022-11-23T01:48:21.1685440Z 
2022-11-23T01:48:21.1685604Z Generating XML reports...
2022-11-23T01:48:21.1686493Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013505.xml
2022-11-23T01:48:21.1686921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1687140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1687561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1687791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1687811Z 
2022-11-23T01:48:21.1687959Z Running tests...
2022-11-23T01:48:21.1688260Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1688625Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1688914Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1689713Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.636s)
2022-11-23T01:48:21.1689858Z 
2022-11-23T01:48:21.1690153Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1690305Z Ran 1 test in 1.636s
2022-11-23T01:48:21.1690325Z 
2022-11-23T01:48:21.1690473Z OK (skipped=1)
2022-11-23T01:48:21.1690492Z 
2022-11-23T01:48:21.1690656Z Generating XML reports...
2022-11-23T01:48:21.1691149Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013513.xml
2022-11-23T01:48:21.1691561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1691780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1692199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1692417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1692455Z 
2022-11-23T01:48:21.1692585Z Running tests...
2022-11-23T01:48:21.1692886Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1693239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1693531Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1693860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15670
2022-11-23T01:48:21.1694129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15671
2022-11-23T01:48:21.1694543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1694741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1695170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1695399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1695807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1696022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1696434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1696664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1696949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1697233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1697663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1698100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1698373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1698641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1698937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnzevs0ca
2022-11-23T01:48:21.1699250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnzevs0ca/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1699542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp202otuc0
2022-11-23T01:48:21.1699847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp202otuc0/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1700047Z ok (5.734s)
2022-11-23T01:48:21.1700067Z 
2022-11-23T01:48:21.1700358Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1700511Z Ran 1 test in 5.735s
2022-11-23T01:48:21.1700531Z 
2022-11-23T01:48:21.1700660Z OK
2022-11-23T01:48:21.1700680Z 
2022-11-23T01:48:21.1700843Z Generating XML reports...
2022-11-23T01:48:21.1701338Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013517.xml
2022-11-23T01:48:21.1701752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1701967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1702386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1702600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1702649Z 
2022-11-23T01:48:21.1702779Z Running tests...
2022-11-23T01:48:21.1703086Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1703437Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1703748Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1704006Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15781
2022-11-23T01:48:21.1704262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15782
2022-11-23T01:48:21.1704716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1704940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1705343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1705580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1705982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1706193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1706603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1706830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1707117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1707401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1707822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1708264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1708533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1708793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1709088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7gz4nc9
2022-11-23T01:48:21.1709394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7gz4nc9/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1709691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1ycjdps
2022-11-23T01:48:21.1710002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1ycjdps/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1710279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1710589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1711039Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpa7gz4nc9'>
2022-11-23T01:48:21.1711244Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T01:48:21.1711681Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpo1ycjdps'>
2022-11-23T01:48:21.1711881Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T01:48:21.1712023Z ok (5.512s)
2022-11-23T01:48:21.1712044Z 
2022-11-23T01:48:21.1712348Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1712500Z Ran 1 test in 5.513s
2022-11-23T01:48:21.1712520Z 
2022-11-23T01:48:21.1712630Z OK
2022-11-23T01:48:21.1712666Z 
2022-11-23T01:48:21.1712811Z Generating XML reports...
2022-11-23T01:48:21.1713307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013525.xml
2022-11-23T01:48:21.1713727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1713943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1714357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1714586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1714606Z 
2022-11-23T01:48:21.1714754Z Running tests...
2022-11-23T01:48:21.1715101Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1715445Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1715749Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1716007Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15896
2022-11-23T01:48:21.1716268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15897
2022-11-23T01:48:21.1716676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1716891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1717307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1717537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1717925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1718139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1718559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1718790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1719074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1719356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1719796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1720229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1720497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1720780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptytfnhhe
2022-11-23T01:48:21.1721090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptytfnhhe/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1721414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1721709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpex9i7ov2
2022-11-23T01:48:21.1722015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpex9i7ov2/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1722290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1722564Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1722708Z ok (4.351s)
2022-11-23T01:48:21.1722729Z 
2022-11-23T01:48:21.1723039Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1723174Z Ran 1 test in 4.351s
2022-11-23T01:48:21.1723194Z 
2022-11-23T01:48:21.1723324Z OK
2022-11-23T01:48:21.1723343Z 
2022-11-23T01:48:21.1723506Z Generating XML reports...
2022-11-23T01:48:21.1723998Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013534.xml
2022-11-23T01:48:21.1724408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1724622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1725036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1725265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1725285Z 
2022-11-23T01:48:21.1725463Z Running tests...
2022-11-23T01:48:21.1725958Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1726331Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1726636Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1726901Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16039
2022-11-23T01:48:21.1727157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16040
2022-11-23T01:48:21.1727568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1727781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1728201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1728418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1728822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1729035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1729448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1729678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1729961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1730241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1730683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1731106Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1731377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1731644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1732026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_u4hz5t
2022-11-23T01:48:21.1732330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_u4hz5t/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1732620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavaum396
2022-11-23T01:48:21.1732927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavaum396/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1733203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1733476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1733602Z ok (5.667s)
2022-11-23T01:48:21.1733621Z 
2022-11-23T01:48:21.1733931Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1734082Z Ran 1 test in 5.668s
2022-11-23T01:48:21.1734102Z 
2022-11-23T01:48:21.1734234Z OK
2022-11-23T01:48:21.1734257Z 
2022-11-23T01:48:21.1734418Z Generating XML reports...
2022-11-23T01:48:21.1734911Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013540.xml
2022-11-23T01:48:21.1735323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1735538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1735941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1736234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1736256Z 
2022-11-23T01:48:21.1736412Z Running tests...
2022-11-23T01:48:21.1736714Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1737065Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1737396Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1737655Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16154
2022-11-23T01:48:21.1737914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16155
2022-11-23T01:48:21.1738326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1738521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1738946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1739176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1739580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1739795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1740209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1740472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1740763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1741048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1741471Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1741908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1742179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1742502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1742783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.1743064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.1743497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1743931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1744212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:21.1744471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:21.1744904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.1745341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.1745641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60m1kwed
2022-11-23T01:48:21.1745952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60m1kwed/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1746243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp453l12qf
2022-11-23T01:48:21.1746606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp453l12qf/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1746757Z ok (5.215s)
2022-11-23T01:48:21.1746777Z 
2022-11-23T01:48:21.1747086Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1747220Z Ran 1 test in 5.216s
2022-11-23T01:48:21.1747240Z 
2022-11-23T01:48:21.1747373Z OK
2022-11-23T01:48:21.1747393Z 
2022-11-23T01:48:21.1747560Z Generating XML reports...
2022-11-23T01:48:21.1748051Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013549.xml
2022-11-23T01:48:21.1748459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1748676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1749099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1749335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1749355Z 
2022-11-23T01:48:21.1749485Z Running tests...
2022-11-23T01:48:21.1749787Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1750143Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1750464Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1750724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16277
2022-11-23T01:48:21.1750981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16278
2022-11-23T01:48:21.1751390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1751605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1752027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1752240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1752651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1752865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1753334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1753568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1753854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1754136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1754578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1754994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1755267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1755534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1755817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.1756096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.1756533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1756960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1757286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:21.1757568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:21.1757978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.1758423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.1758724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbjazytw
2022-11-23T01:48:21.1759033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbjazytw/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1759325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9mn7ms_k
2022-11-23T01:48:21.1759637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9mn7ms_k/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1759783Z ok (15.069s)
2022-11-23T01:48:21.1759804Z 
2022-11-23T01:48:21.1760110Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1760263Z Ran 1 test in 15.069s
2022-11-23T01:48:21.1760282Z 
2022-11-23T01:48:21.1760394Z OK
2022-11-23T01:48:21.1760413Z 
2022-11-23T01:48:21.1760576Z Generating XML reports...
2022-11-23T01:48:21.1761071Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013556.xml
2022-11-23T01:48:21.1761482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1761696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1762112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1762341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1762365Z 
2022-11-23T01:48:21.1762513Z Running tests...
2022-11-23T01:48:21.1762812Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1763150Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1763756Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1764133Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16400
2022-11-23T01:48:21.1764389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16401
2022-11-23T01:48:21.1764809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1765025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1765450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1765871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1766275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1766489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1766909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1767141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1767426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1767710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1768150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1768660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1768941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1769192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1769490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9_a_xkf
2022-11-23T01:48:21.1769801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9_a_xkf/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1770089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_k2vqemc
2022-11-23T01:48:21.1770391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_k2vqemc/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1770532Z ok (5.645s)
2022-11-23T01:48:21.1770554Z 
2022-11-23T01:48:21.1770867Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1771019Z Ran 1 test in 5.645s
2022-11-23T01:48:21.1771039Z 
2022-11-23T01:48:21.1771149Z OK
2022-11-23T01:48:21.1771187Z 
2022-11-23T01:48:21.1771330Z Generating XML reports...
2022-11-23T01:48:21.1771825Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013614.xml
2022-11-23T01:48:21.1772240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1772457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1772872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1773098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1773118Z 
2022-11-23T01:48:21.1773265Z Running tests...
2022-11-23T01:48:21.1773570Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1773907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1774236Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1774567Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16515
2022-11-23T01:48:21.1774825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16516
2022-11-23T01:48:21.1775237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1775451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1775868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1776097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1776503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1776698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1777114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1777348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1777634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1777915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1778350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1778827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1779104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1779355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1779657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbr2gh12
2022-11-23T01:48:21.1779970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbr2gh12/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1780259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlkjnw7n
2022-11-23T01:48:21.1780566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlkjnw7n/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1780708Z ok (5.748s)
2022-11-23T01:48:21.1780728Z 
2022-11-23T01:48:21.1781035Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1781188Z Ran 1 test in 5.748s
2022-11-23T01:48:21.1781208Z 
2022-11-23T01:48:21.1781342Z OK
2022-11-23T01:48:21.1781362Z 
2022-11-23T01:48:21.1781507Z Generating XML reports...
2022-11-23T01:48:21.1781999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013622.xml
2022-11-23T01:48:21.1782406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1782625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1783041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1783269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1783289Z 
2022-11-23T01:48:21.1783435Z Running tests...
2022-11-23T01:48:21.1783736Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1784074Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1784371Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1784628Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16630
2022-11-23T01:48:21.1784880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16631
2022-11-23T01:48:21.1785347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1785561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1785980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1786209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1786613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1786814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1787231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1787462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1787749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1788030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1788470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1788902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1789173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1789485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1789772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vgnh1ht
2022-11-23T01:48:21.1790082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vgnh1ht/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1790381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphc81ib3r
2022-11-23T01:48:21.1790685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphc81ib3r/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1790827Z ok (5.669s)
2022-11-23T01:48:21.1790847Z 
2022-11-23T01:48:21.1791155Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1791307Z Ran 1 test in 5.669s
2022-11-23T01:48:21.1791326Z 
2022-11-23T01:48:21.1791457Z OK
2022-11-23T01:48:21.1791477Z 
2022-11-23T01:48:21.1791621Z Generating XML reports...
2022-11-23T01:48:21.1792119Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013631.xml
2022-11-23T01:48:21.1792530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1792744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1793163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1793393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1793414Z 
2022-11-23T01:48:21.1793563Z Running tests...
2022-11-23T01:48:21.1793861Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1794212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1794497Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1794757Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16741
2022-11-23T01:48:21.1795012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16742
2022-11-23T01:48:21.1795423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1795697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1796121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1796356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1796755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1796950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1797371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1797601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1797885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1798163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1798605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1799036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1799304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1799571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1799938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59jxkxxf
2022-11-23T01:48:21.1800260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59jxkxxf/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1800552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8bq0gmy
2022-11-23T01:48:21.1800859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8bq0gmy/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1801679Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:21.1802498Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:21.1802779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1803053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1803194Z ok (5.542s)
2022-11-23T01:48:21.1803214Z 
2022-11-23T01:48:21.1803529Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1803684Z Ran 1 test in 5.542s
2022-11-23T01:48:21.1803708Z 
2022-11-23T01:48:21.1803838Z OK
2022-11-23T01:48:21.1803858Z 
2022-11-23T01:48:21.1804002Z Generating XML reports...
2022-11-23T01:48:21.1804493Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013639.xml
2022-11-23T01:48:21.1804905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1805174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1805599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1806012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1806033Z 
2022-11-23T01:48:21.1806188Z Running tests...
2022-11-23T01:48:21.1806494Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1806854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1807159Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1807959Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.636s)
2022-11-23T01:48:21.1807985Z 
2022-11-23T01:48:21.1808288Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1808440Z Ran 1 test in 1.637s
2022-11-23T01:48:21.1808460Z 
2022-11-23T01:48:21.1808607Z OK (skipped=1)
2022-11-23T01:48:21.1808626Z 
2022-11-23T01:48:21.1808788Z Generating XML reports...
2022-11-23T01:48:21.1809278Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013647.xml
2022-11-23T01:48:21.1809760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1809983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1810390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1810628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1810648Z 
2022-11-23T01:48:21.1810799Z Running tests...
2022-11-23T01:48:21.1811104Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1811460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1811777Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1812566Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.650s)
2022-11-23T01:48:21.1812587Z 
2022-11-23T01:48:21.1812887Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1813046Z Ran 1 test in 1.651s
2022-11-23T01:48:21.1813065Z 
2022-11-23T01:48:21.1813213Z OK (skipped=1)
2022-11-23T01:48:21.1813232Z 
2022-11-23T01:48:21.1813376Z Generating XML reports...
2022-11-23T01:48:21.1813864Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013651.xml
2022-11-23T01:48:21.1814277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1814492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1814914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1815144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1815164Z 
2022-11-23T01:48:21.1815310Z Running tests...
2022-11-23T01:48:21.1815612Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1816041Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1816340Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1816598Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16924
2022-11-23T01:48:21.1816856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16925
2022-11-23T01:48:21.1817266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1817483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1817899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1818128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1818537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1818732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1819143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1819372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1819655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1819998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1820445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1820880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1821152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1821420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1821698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmproqqmd8b
2022-11-23T01:48:21.1822008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmproqqmd8b/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1822299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe79h5ncs
2022-11-23T01:48:21.1822607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe79h5ncs/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1822985Z STAGE:2022-11-23 01:37:00 16925:16925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1823352Z STAGE:2022-11-23 01:37:00 16924:16924 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1823635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1823908Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1824287Z STAGE:2022-11-23 01:37:01 16924:16924 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1824639Z STAGE:2022-11-23 01:37:01 16925:16925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1825026Z STAGE:2022-11-23 01:37:01 16924:16924 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1825414Z STAGE:2022-11-23 01:37:01 16925:16925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1826235Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:21.1827114Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:21.1827698Z STAGE:2022-11-23 01:37:01 16924:16924 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:37:01 16925:16925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1827723Z 
2022-11-23T01:48:21.1828099Z STAGE:2022-11-23 01:37:01 16925:16925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1828467Z STAGE:2022-11-23 01:37:01 16924:16924 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1828850Z STAGE:2022-11-23 01:37:01 16925:16925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1829225Z STAGE:2022-11-23 01:37:01 16924:16924 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1829367Z ok (6.044s)
2022-11-23T01:48:21.1829387Z 
2022-11-23T01:48:21.1829732Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1829891Z Ran 1 test in 6.044s
2022-11-23T01:48:21.1829911Z 
2022-11-23T01:48:21.1830022Z OK
2022-11-23T01:48:21.1830041Z 
2022-11-23T01:48:21.1830206Z Generating XML reports...
2022-11-23T01:48:21.1830700Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013655.xml
2022-11-23T01:48:21.1831118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1831332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1831750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1831978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1831999Z 
2022-11-23T01:48:21.1832149Z Running tests...
2022-11-23T01:48:21.1832434Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1832787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1833093Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1833353Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17043
2022-11-23T01:48:21.1833608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17044
2022-11-23T01:48:21.1834016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1834230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1834645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1834879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1835263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1835477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1835894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1836177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1836461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1836745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1837187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1837626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1837901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1838149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1838446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjs5s4ueh
2022-11-23T01:48:21.1838761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjs5s4ueh/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1839051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_nnfeh3
2022-11-23T01:48:21.1839352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_nnfeh3/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1839494Z ok (5.197s)
2022-11-23T01:48:21.1839515Z 
2022-11-23T01:48:21.1839822Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1840020Z Ran 1 test in 5.197s
2022-11-23T01:48:21.1840042Z 
2022-11-23T01:48:21.1840154Z OK
2022-11-23T01:48:21.1840173Z 
2022-11-23T01:48:21.1840340Z Generating XML reports...
2022-11-23T01:48:21.1840880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013704.xml
2022-11-23T01:48:21.1841296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1841510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1841925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1842155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1842175Z 
2022-11-23T01:48:21.1842321Z Running tests...
2022-11-23T01:48:21.1842620Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1842961Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1843275Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1844064Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.646s)
2022-11-23T01:48:21.1844089Z 
2022-11-23T01:48:21.1844391Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1844542Z Ran 1 test in 1.647s
2022-11-23T01:48:21.1844562Z 
2022-11-23T01:48:21.1844710Z OK (skipped=1)
2022-11-23T01:48:21.1844729Z 
2022-11-23T01:48:21.1844890Z Generating XML reports...
2022-11-23T01:48:21.1845378Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013712.xml
2022-11-23T01:48:21.1845945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1846150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1846664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1846985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1847005Z 
2022-11-23T01:48:21.1847153Z Running tests...
2022-11-23T01:48:21.1847455Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1847807Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1848128Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1848389Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17188
2022-11-23T01:48:21.1848647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17189
2022-11-23T01:48:21.1849041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1849258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1849682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1849910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1850309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1850525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1851164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1851400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1851667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1851950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1852402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1852836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1853106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1853374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1853668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3zb7c1tp
2022-11-23T01:48:21.1853979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3zb7c1tp/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1854275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1wc2w8p
2022-11-23T01:48:21.1854563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1wc2w8p/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1855521Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T01:48:21.1855677Z   warnings.warn(
2022-11-23T01:48:21.1856627Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T01:48:21.1856780Z   warnings.warn(
2022-11-23T01:48:21.1857057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1857380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1857523Z ok (5.554s)
2022-11-23T01:48:21.1857543Z 
2022-11-23T01:48:21.1857849Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1858003Z Ran 1 test in 5.554s
2022-11-23T01:48:21.1858022Z 
2022-11-23T01:48:21.1858135Z OK
2022-11-23T01:48:21.1858177Z 
2022-11-23T01:48:21.1858323Z Generating XML reports...
2022-11-23T01:48:21.1858817Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013716.xml
2022-11-23T01:48:21.1859229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1859444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1859860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1860094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1860114Z 
2022-11-23T01:48:21.1860263Z Running tests...
2022-11-23T01:48:21.1860566Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1860902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1861219Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1862056Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.646s)
2022-11-23T01:48:21.1862079Z 
2022-11-23T01:48:21.1862384Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1862541Z Ran 1 test in 1.647s
2022-11-23T01:48:21.1862561Z 
2022-11-23T01:48:21.1862707Z OK (skipped=1)
2022-11-23T01:48:21.1862726Z 
2022-11-23T01:48:21.1862888Z Generating XML reports...
2022-11-23T01:48:21.1863378Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013724.xml
2022-11-23T01:48:21.1863786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1864004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1864410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1864641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1864661Z 
2022-11-23T01:48:21.1864813Z Running tests...
2022-11-23T01:48:21.1865113Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1865472Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1865781Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1866039Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17337
2022-11-23T01:48:21.1866296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17338
2022-11-23T01:48:21.1866687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1866909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1867329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1867557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1868015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1868228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1868638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1868867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1869151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1869420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1869860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1870299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1870572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1870836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1871129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2zqkv1g
2022-11-23T01:48:21.1871441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2zqkv1g/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1871732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9xgky7d
2022-11-23T01:48:21.1872083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9xgky7d/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1872453Z STAGE:2022-11-23 01:37:33 17338:17338 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1872825Z STAGE:2022-11-23 01:37:33 17337:17337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1873107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1873382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:48:21.1873754Z STAGE:2022-11-23 01:37:33 17338:17338 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1874118Z STAGE:2022-11-23 01:37:33 17337:17337 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1874724Z STAGE:2022-11-23 01:37:33 17338:17338 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:37:33 17337:17337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1874745Z 
2022-11-23T01:48:21.1875107Z STAGE:2022-11-23 01:37:33 17337:17337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1875475Z STAGE:2022-11-23 01:37:34 17337:17337 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1875847Z STAGE:2022-11-23 01:37:34 17337:17337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1875996Z ok (6.118s)
2022-11-23T01:48:21.1876017Z 
2022-11-23T01:48:21.1876319Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1876470Z Ran 1 test in 6.118s
2022-11-23T01:48:21.1876490Z 
2022-11-23T01:48:21.1876620Z OK
2022-11-23T01:48:21.1876639Z 
2022-11-23T01:48:21.1876801Z Generating XML reports...
2022-11-23T01:48:21.1877291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013728.xml
2022-11-23T01:48:21.1877706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1877921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1878322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1878610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1878629Z 
2022-11-23T01:48:21.1878777Z Running tests...
2022-11-23T01:48:21.1879080Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1879432Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1879736Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1879994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17456
2022-11-23T01:48:21.1880254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17457
2022-11-23T01:48:21.1880651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1880867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1881289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1881518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1881921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1882132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1882550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1882821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1883114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1883379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1883827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1884262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1884531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1884797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1885095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbzonzsjh
2022-11-23T01:48:21.1885406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbzonzsjh/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1885863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_vwg1rx
2022-11-23T01:48:21.1886177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_vwg1rx/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1886307Z ok (5.107s)
2022-11-23T01:48:21.1886328Z 
2022-11-23T01:48:21.1886637Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1886792Z Ran 1 test in 5.108s
2022-11-23T01:48:21.1886812Z 
2022-11-23T01:48:21.1886946Z OK
2022-11-23T01:48:21.1886966Z 
2022-11-23T01:48:21.1887128Z Generating XML reports...
2022-11-23T01:48:21.1887620Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013737.xml
2022-11-23T01:48:21.1888032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1888252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1888656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1888889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1888981Z 
2022-11-23T01:48:21.1889138Z Running tests...
2022-11-23T01:48:21.1889441Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1889796Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1890111Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1890366Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17567
2022-11-23T01:48:21.1890620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17568
2022-11-23T01:48:21.1891036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1891235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1891660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1891897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1892300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1892511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1892927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1893153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1893510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1893784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1894228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1894668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1894936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1895201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1895498Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrknwz23
2022-11-23T01:48:21.1895809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrknwz23/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1896106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo14qjeg0
2022-11-23T01:48:21.1896410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo14qjeg0/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1896535Z ok (5.081s)
2022-11-23T01:48:21.1896555Z 
2022-11-23T01:48:21.1896859Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1897018Z Ran 1 test in 5.082s
2022-11-23T01:48:21.1897038Z 
2022-11-23T01:48:21.1897168Z OK
2022-11-23T01:48:21.1897188Z 
2022-11-23T01:48:21.1897347Z Generating XML reports...
2022-11-23T01:48:21.1897838Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013744.xml
2022-11-23T01:48:21.1898248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1898463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1898865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1899100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1899119Z 
2022-11-23T01:48:21.1899267Z Running tests...
2022-11-23T01:48:21.1899570Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1899981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1900295Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1901083Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.656s)
2022-11-23T01:48:21.1901104Z 
2022-11-23T01:48:21.1901401Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1901556Z Ran 1 test in 1.656s
2022-11-23T01:48:21.1901575Z 
2022-11-23T01:48:21.1901724Z OK (skipped=1)
2022-11-23T01:48:21.1901743Z 
2022-11-23T01:48:21.1901889Z Generating XML reports...
2022-11-23T01:48:21.1902382Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013752.xml
2022-11-23T01:48:21.1902797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1903012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1903431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1903660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1903680Z 
2022-11-23T01:48:21.1903826Z Running tests...
2022-11-23T01:48:21.1904180Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1904521Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1904819Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1905608Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.654s)
2022-11-23T01:48:21.1905629Z 
2022-11-23T01:48:21.1905933Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1906083Z Ran 1 test in 1.654s
2022-11-23T01:48:21.1906103Z 
2022-11-23T01:48:21.1906249Z OK (skipped=1)
2022-11-23T01:48:21.1906268Z 
2022-11-23T01:48:21.1906483Z Generating XML reports...
2022-11-23T01:48:21.1906972Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013756.xml
2022-11-23T01:48:21.1907382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1907598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1908002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1908234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1908253Z 
2022-11-23T01:48:21.1908402Z Running tests...
2022-11-23T01:48:21.1908705Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1909058Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1909387Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1910176Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.644s)
2022-11-23T01:48:21.1910247Z 
2022-11-23T01:48:21.1910550Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1910702Z Ran 1 test in 1.644s
2022-11-23T01:48:21.1910722Z 
2022-11-23T01:48:21.1910868Z OK (skipped=1)
2022-11-23T01:48:21.1910887Z 
2022-11-23T01:48:21.1911030Z Generating XML reports...
2022-11-23T01:48:21.1911521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013800.xml
2022-11-23T01:48:21.1911928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1912146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1912563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1912790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1912813Z 
2022-11-23T01:48:21.1912961Z Running tests...
2022-11-23T01:48:21.1913262Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1913597Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1913930Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1914189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17780
2022-11-23T01:48:21.1914445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17781
2022-11-23T01:48:21.1914899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1915122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1915542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1915776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1916183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1916379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1916791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1917019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1917305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1917587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1918029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1918475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1918745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1919010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1919288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_v0y__d
2022-11-23T01:48:21.1919594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_v0y__d/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1919893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmad8rpm9
2022-11-23T01:48:21.1920199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmad8rpm9/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1920339Z ok (5.551s)
2022-11-23T01:48:21.1920360Z 
2022-11-23T01:48:21.1920664Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1920867Z Ran 1 test in 5.551s
2022-11-23T01:48:21.1920887Z 
2022-11-23T01:48:21.1921016Z OK
2022-11-23T01:48:21.1921035Z 
2022-11-23T01:48:21.1921180Z Generating XML reports...
2022-11-23T01:48:21.1921670Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013805.xml
2022-11-23T01:48:21.1922079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1922294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1922717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1922946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1922966Z 
2022-11-23T01:48:21.1923114Z Running tests...
2022-11-23T01:48:21.1923415Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1923772Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1924063Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1924321Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17895
2022-11-23T01:48:21.1924576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17896
2022-11-23T01:48:21.1924985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1925241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1925889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1926132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1926547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1926742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1927157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1927385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1927670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1927953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1928396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1928833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1929111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1929379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1929782Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T01:48:21.1930076Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T01:48:21.1930487Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T01:48:21.1930776Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T01:48:21.1931071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpioogqgwk
2022-11-23T01:48:21.1931381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpioogqgwk/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1931756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qpymhtm
2022-11-23T01:48:21.1932065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qpymhtm/_remote_module_non_scriptable.py
2022-11-23T01:48:21.1932208Z ok (5.074s)
2022-11-23T01:48:21.1932228Z 
2022-11-23T01:48:21.1932519Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1932670Z Ran 1 test in 5.074s
2022-11-23T01:48:21.1932690Z 
2022-11-23T01:48:21.1932823Z OK
2022-11-23T01:48:21.1932843Z 
2022-11-23T01:48:21.1933003Z Generating XML reports...
2022-11-23T01:48:21.1933497Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013813.xml
2022-11-23T01:48:21.1933908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1934122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1934538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1934755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1934792Z 
2022-11-23T01:48:21.1934922Z Running tests...
2022-11-23T01:48:21.1935224Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1935576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1935876Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1936192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18006
2022-11-23T01:48:21.1936458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18007
2022-11-23T01:48:21.1936875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1937094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1937500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1937728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1938132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1938343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1938759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1938986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1939269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1939549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1939979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1940416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1940727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1940998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1941281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.1941563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.1942000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1942486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1942630Z ok (4.196s)
2022-11-23T01:48:21.1942650Z 
2022-11-23T01:48:21.1942935Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1943088Z Ran 1 test in 4.196s
2022-11-23T01:48:21.1943108Z 
2022-11-23T01:48:21.1943239Z OK
2022-11-23T01:48:21.1943258Z 
2022-11-23T01:48:21.1943420Z Generating XML reports...
2022-11-23T01:48:21.1943909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013820.xml
2022-11-23T01:48:21.1944319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1944533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1944949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1945183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1945203Z 
2022-11-23T01:48:21.1945335Z Running tests...
2022-11-23T01:48:21.1945640Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1945992Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1946280Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1946535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18121
2022-11-23T01:48:21.1946835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18122
2022-11-23T01:48:21.1947252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1947465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1947867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1948099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1948506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1948717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1949129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1949358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1949639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1949918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1950356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1950780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1951049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1951320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1951596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.1951876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.1952306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1952735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.1952932Z ok (4.240s)
2022-11-23T01:48:21.1952952Z 
2022-11-23T01:48:21.1953255Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1953391Z Ran 1 test in 4.240s
2022-11-23T01:48:21.1953410Z 
2022-11-23T01:48:21.1953541Z OK
2022-11-23T01:48:21.1953560Z 
2022-11-23T01:48:21.1953720Z Generating XML reports...
2022-11-23T01:48:21.1954211Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013827.xml
2022-11-23T01:48:21.1954623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1954836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1955252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1955484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1955508Z 
2022-11-23T01:48:21.1955638Z Running tests...
2022-11-23T01:48:21.1955940Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1956288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1956601Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1957433Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.630s)
2022-11-23T01:48:21.1957456Z 
2022-11-23T01:48:21.1957758Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1957909Z Ran 1 test in 1.631s
2022-11-23T01:48:21.1957929Z 
2022-11-23T01:48:21.1958080Z OK (skipped=1)
2022-11-23T01:48:21.1958099Z 
2022-11-23T01:48:21.1958260Z Generating XML reports...
2022-11-23T01:48:21.1958753Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013834.xml
2022-11-23T01:48:21.1959148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1959364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1959782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1960013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1960032Z 
2022-11-23T01:48:21.1960180Z Running tests...
2022-11-23T01:48:21.1960477Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1960827Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1961146Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1961926Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.625s)
2022-11-23T01:48:21.1961947Z 
2022-11-23T01:48:21.1962251Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1962387Z Ran 1 test in 1.626s
2022-11-23T01:48:21.1962411Z 
2022-11-23T01:48:21.1962560Z OK (skipped=1)
2022-11-23T01:48:21.1962580Z 
2022-11-23T01:48:21.1962742Z Generating XML reports...
2022-11-23T01:48:21.1963225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013838.xml
2022-11-23T01:48:21.1963736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1963949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1964366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1964595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1964615Z 
2022-11-23T01:48:21.1964744Z Running tests...
2022-11-23T01:48:21.1965043Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1965395Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1965929Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1966199Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18304
2022-11-23T01:48:21.1966458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18305
2022-11-23T01:48:21.1966871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1967086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1967504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1967716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1968204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1968429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1968844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1969073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1969362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1969643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1970083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1970500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1970774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1971042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1971182Z ok (4.350s)
2022-11-23T01:48:21.1971203Z 
2022-11-23T01:48:21.1971506Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1971661Z Ran 1 test in 4.350s
2022-11-23T01:48:21.1971681Z 
2022-11-23T01:48:21.1971811Z OK
2022-11-23T01:48:21.1971831Z 
2022-11-23T01:48:21.1971992Z Generating XML reports...
2022-11-23T01:48:21.1972478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013842.xml
2022-11-23T01:48:21.1972869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1973081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1973501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1973733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1973753Z 
2022-11-23T01:48:21.1973899Z Running tests...
2022-11-23T01:48:21.1974197Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1974615Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1974895Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1975134Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18413
2022-11-23T01:48:21.1975394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18414
2022-11-23T01:48:21.1975802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1976019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1976433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1976661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1977064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1977280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1977689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1977901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1978185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1978468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1978956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1979400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1979665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1980043Z STAGE:2022-11-23 01:38:53 18414:18414 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1980310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1980684Z STAGE:2022-11-23 01:38:53 18413:18413 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1981041Z STAGE:2022-11-23 01:38:53 18414:18414 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1981434Z STAGE:2022-11-23 01:38:53 18414:18414 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1981807Z STAGE:2022-11-23 01:38:53 18413:18413 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1982187Z STAGE:2022-11-23 01:38:53 18413:18413 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1982547Z STAGE:2022-11-23 01:38:53 18414:18414 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1982907Z STAGE:2022-11-23 01:38:53 18413:18413 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.1983281Z STAGE:2022-11-23 01:38:53 18413:18413 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1983660Z STAGE:2022-11-23 01:38:53 18413:18413 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1984033Z STAGE:2022-11-23 01:38:53 18414:18414 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.1984402Z STAGE:2022-11-23 01:38:53 18414:18414 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.1984545Z ok (4.319s)
2022-11-23T01:48:21.1984565Z 
2022-11-23T01:48:21.1984869Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1985021Z Ran 1 test in 4.320s
2022-11-23T01:48:21.1985041Z 
2022-11-23T01:48:21.1985222Z OK
2022-11-23T01:48:21.1985242Z 
2022-11-23T01:48:21.1985404Z Generating XML reports...
2022-11-23T01:48:21.1985899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013849.xml
2022-11-23T01:48:21.1986309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1986504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1986926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1987160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1987180Z 
2022-11-23T01:48:21.1987327Z Running tests...
2022-11-23T01:48:21.1987628Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1987980Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1988276Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.1988535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18526
2022-11-23T01:48:21.1988794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18527
2022-11-23T01:48:21.1989186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1989399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1989868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1990104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1990508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1990718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1991133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1991362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1991627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.1991910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.1992355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1992792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.1993061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.1993326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.1993471Z ok (4.274s)
2022-11-23T01:48:21.1993492Z 
2022-11-23T01:48:21.1993795Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1993945Z Ran 1 test in 4.274s
2022-11-23T01:48:21.1993966Z 
2022-11-23T01:48:21.1994077Z OK
2022-11-23T01:48:21.1994096Z 
2022-11-23T01:48:21.1994256Z Generating XML reports...
2022-11-23T01:48:21.1994748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013856.xml
2022-11-23T01:48:21.1995159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1995371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1995786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1996076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1996096Z 
2022-11-23T01:48:21.1996243Z Running tests...
2022-11-23T01:48:21.1996527Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1996883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.1997172Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T01:48:21.1997193Z 
2022-11-23T01:48:21.1997491Z ----------------------------------------------------------------------
2022-11-23T01:48:21.1997639Z Ran 1 test in 0.002s
2022-11-23T01:48:21.1997662Z 
2022-11-23T01:48:21.1997809Z OK (skipped=1)
2022-11-23T01:48:21.1997828Z 
2022-11-23T01:48:21.1997990Z Generating XML reports...
2022-11-23T01:48:21.1998473Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013903.xml
2022-11-23T01:48:21.1998882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.1999076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.1999494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.1999724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.1999744Z 
2022-11-23T01:48:21.1999891Z Running tests...
2022-11-23T01:48:21.2000187Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2000584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2000886Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2001143Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18668
2022-11-23T01:48:21.2001401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18669
2022-11-23T01:48:21.2001795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2002009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2002428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2002656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2003062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2003274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2003685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2003912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2004185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2004465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2004904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2005334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2005607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2006091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2006378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2006661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2007184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2007599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2007975Z STAGE:2022-11-23 01:39:09 18668:18668 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2008338Z STAGE:2022-11-23 01:39:09 18669:18669 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2008718Z STAGE:2022-11-23 01:39:09 18669:18669 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2009106Z STAGE:2022-11-23 01:39:09 18669:18669 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2009477Z STAGE:2022-11-23 01:39:09 18668:18668 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2009866Z STAGE:2022-11-23 01:39:09 18668:18668 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2010232Z STAGE:2022-11-23 01:39:09 18669:18669 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2010598Z STAGE:2022-11-23 01:39:09 18668:18668 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2010952Z STAGE:2022-11-23 01:39:09 18668:18668 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2011336Z STAGE:2022-11-23 01:39:09 18668:18668 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2011764Z STAGE:2022-11-23 01:39:09 18669:18669 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2012158Z STAGE:2022-11-23 01:39:09 18669:18669 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2012303Z ok (4.365s)
2022-11-23T01:48:21.2012325Z 
2022-11-23T01:48:21.2012632Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2012785Z Ran 1 test in 4.365s
2022-11-23T01:48:21.2012805Z 
2022-11-23T01:48:21.2012932Z OK
2022-11-23T01:48:21.2012951Z 
2022-11-23T01:48:21.2013112Z Generating XML reports...
2022-11-23T01:48:21.2013588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013905.xml
2022-11-23T01:48:21.2013998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2014213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2014637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2014868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2014888Z 
2022-11-23T01:48:21.2015033Z Running tests...
2022-11-23T01:48:21.2015339Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2015689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2015963Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2016222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18787
2022-11-23T01:48:21.2016477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18788
2022-11-23T01:48:21.2016885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2017102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2017518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2017749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2018205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2018416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2018810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2019042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2019325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2019610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2020047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2020478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2020752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2021018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2021217Z skip: Skipped due to small world size. (4.249s)
2022-11-23T01:48:21.2021237Z 
2022-11-23T01:48:21.2021525Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2021679Z Ran 1 test in 4.249s
2022-11-23T01:48:21.2021699Z 
2022-11-23T01:48:21.2021844Z OK (skipped=1)
2022-11-23T01:48:21.2021863Z 
2022-11-23T01:48:21.2022026Z Generating XML reports...
2022-11-23T01:48:21.2022561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013912.xml
2022-11-23T01:48:21.2022981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2023200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2023618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2023829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2023867Z 
2022-11-23T01:48:21.2023998Z Running tests...
2022-11-23T01:48:21.2024298Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2024651Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2024944Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2025201Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18896
2022-11-23T01:48:21.2025456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18897
2022-11-23T01:48:21.2025863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2026081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2026481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2026710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2027114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2027326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2027739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2027965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2028250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2028586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2029011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2029446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2029714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2029980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2030125Z ok (4.311s)
2022-11-23T01:48:21.2030146Z 
2022-11-23T01:48:21.2030447Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2030596Z Ran 1 test in 4.311s
2022-11-23T01:48:21.2030615Z 
2022-11-23T01:48:21.2030744Z OK
2022-11-23T01:48:21.2030763Z 
2022-11-23T01:48:21.2030927Z Generating XML reports...
2022-11-23T01:48:21.2031398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013919.xml
2022-11-23T01:48:21.2031811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2032027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2032446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2032674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2032746Z 
2022-11-23T01:48:21.2032901Z Running tests...
2022-11-23T01:48:21.2033202Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2033554Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2033840Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2034637Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.653s)
2022-11-23T01:48:21.2034677Z 
2022-11-23T01:48:21.2034957Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2035107Z Ran 1 test in 1.654s
2022-11-23T01:48:21.2035126Z 
2022-11-23T01:48:21.2035269Z OK (skipped=1)
2022-11-23T01:48:21.2035291Z 
2022-11-23T01:48:21.2035451Z Generating XML reports...
2022-11-23T01:48:21.2035938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013926.xml
2022-11-23T01:48:21.2036343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2036561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2036975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2037188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2037225Z 
2022-11-23T01:48:21.2037356Z Running tests...
2022-11-23T01:48:21.2037658Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2038008Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2038300Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2038558Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19039
2022-11-23T01:48:21.2038812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19040
2022-11-23T01:48:21.2039281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2039495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2039896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2040125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2040563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2040781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2041199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2041427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2041710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2041997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2042418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2042853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2043120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2043442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2043730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2044010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2044457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2044887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2045030Z ok (4.426s)
2022-11-23T01:48:21.2045050Z 
2022-11-23T01:48:21.2045332Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2045485Z Ran 1 test in 4.426s
2022-11-23T01:48:21.2045505Z 
2022-11-23T01:48:21.2045634Z OK
2022-11-23T01:48:21.2045654Z 
2022-11-23T01:48:21.2046025Z Generating XML reports...
2022-11-23T01:48:21.2046530Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013930.xml
2022-11-23T01:48:21.2046946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2047163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2047588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2047819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2047839Z 
2022-11-23T01:48:21.2047970Z Running tests...
2022-11-23T01:48:21.2048268Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2048622Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2048909Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2049167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19154
2022-11-23T01:48:21.2049417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19155
2022-11-23T01:48:21.2049826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2050121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2050527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2050759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2051160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2051371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2051790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2052015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2052298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2052583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2053022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2053438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2053708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2053971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2054110Z ok (4.262s)
2022-11-23T01:48:21.2054190Z 
2022-11-23T01:48:21.2054505Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2054658Z Ran 1 test in 4.263s
2022-11-23T01:48:21.2054678Z 
2022-11-23T01:48:21.2054808Z OK
2022-11-23T01:48:21.2054827Z 
2022-11-23T01:48:21.2054989Z Generating XML reports...
2022-11-23T01:48:21.2055464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013937.xml
2022-11-23T01:48:21.2055875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2056090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2056504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2056731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2056750Z 
2022-11-23T01:48:21.2056904Z Running tests...
2022-11-23T01:48:21.2057205Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2057559Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2057840Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2058082Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19263
2022-11-23T01:48:21.2058339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19264
2022-11-23T01:48:21.2058748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2058962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2059378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2059608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2060007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2060219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2060610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2060893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2061177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2061456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2061898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2062333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2062601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2062867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2063011Z ok (4.217s)
2022-11-23T01:48:21.2063031Z 
2022-11-23T01:48:21.2063314Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2063465Z Ran 1 test in 4.217s
2022-11-23T01:48:21.2063485Z 
2022-11-23T01:48:21.2063612Z OK
2022-11-23T01:48:21.2063631Z 
2022-11-23T01:48:21.2063793Z Generating XML reports...
2022-11-23T01:48:21.2064280Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013944.xml
2022-11-23T01:48:21.2064689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2064950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2065379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2065610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2065634Z 
2022-11-23T01:48:21.2065765Z Running tests...
2022-11-23T01:48:21.2066068Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2066417Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2066720Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2066978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19372
2022-11-23T01:48:21.2067234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19373
2022-11-23T01:48:21.2067645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2067859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2068260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2068492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2068894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2069104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2069513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2069736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2070021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2070303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2070743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2071163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2071488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2071755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2072033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2072311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2072751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2073184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2073326Z ok (4.360s)
2022-11-23T01:48:21.2073346Z 
2022-11-23T01:48:21.2073651Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2073790Z Ran 1 test in 4.360s
2022-11-23T01:48:21.2073810Z 
2022-11-23T01:48:21.2073942Z OK
2022-11-23T01:48:21.2073961Z 
2022-11-23T01:48:21.2074123Z Generating XML reports...
2022-11-23T01:48:21.2074612Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013951.xml
2022-11-23T01:48:21.2075021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2075235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2075694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2075930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2075950Z 
2022-11-23T01:48:21.2076079Z Running tests...
2022-11-23T01:48:21.2076384Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2076742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2077041Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2077295Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19487
2022-11-23T01:48:21.2077549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19488
2022-11-23T01:48:21.2077958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2078175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2078591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2078804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2079215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2079426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2079833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2080061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2080346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2080632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2081068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2081485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2081812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2082082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2082356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2082632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2083070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2083503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2083646Z ok (4.428s)
2022-11-23T01:48:21.2083666Z 
2022-11-23T01:48:21.2083968Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2084106Z Ran 1 test in 4.428s
2022-11-23T01:48:21.2084126Z 
2022-11-23T01:48:21.2084255Z OK
2022-11-23T01:48:21.2084275Z 
2022-11-23T01:48:21.2084435Z Generating XML reports...
2022-11-23T01:48:21.2084923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013957.xml
2022-11-23T01:48:21.2085334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2085548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2086260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2086501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2086523Z 
2022-11-23T01:48:21.2086670Z Running tests...
2022-11-23T01:48:21.2086958Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2087316Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2087616Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2087874Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19602
2022-11-23T01:48:21.2088130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19603
2022-11-23T01:48:21.2088542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2088756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2089164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2089359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2089778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2090011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2090424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2090652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2090935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2091219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2091659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2092093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2092343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2092679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2092978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbqfw7ss8
2022-11-23T01:48:21.2093286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbqfw7ss8/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2093578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_putqov
2022-11-23T01:48:21.2093878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_putqov/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2094024Z ok (5.547s)
2022-11-23T01:48:21.2094045Z 
2022-11-23T01:48:21.2094353Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2094486Z Ran 1 test in 5.547s
2022-11-23T01:48:21.2094525Z 
2022-11-23T01:48:21.2094637Z OK
2022-11-23T01:48:21.2094656Z 
2022-11-23T01:48:21.2094821Z Generating XML reports...
2022-11-23T01:48:21.2095311Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014004.xml
2022-11-23T01:48:21.2095724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2095937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2096358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2096586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2096653Z 
2022-11-23T01:48:21.2096805Z Running tests...
2022-11-23T01:48:21.2097086Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2097439Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2097714Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2097974Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19717
2022-11-23T01:48:21.2098228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19718
2022-11-23T01:48:21.2098637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2098851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2099269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2099485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2099891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2100103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2100523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2100752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2101031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2101311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2101744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2102183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2102435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2102701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2102902Z ok (4.279s)
2022-11-23T01:48:21.2102922Z 
2022-11-23T01:48:21.2103228Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2103381Z Ran 1 test in 4.279s
2022-11-23T01:48:21.2103401Z 
2022-11-23T01:48:21.2103534Z OK
2022-11-23T01:48:21.2103554Z 
2022-11-23T01:48:21.2103717Z Generating XML reports...
2022-11-23T01:48:21.2104207Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014013.xml
2022-11-23T01:48:21.2104623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2104823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2105238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2105467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2105490Z 
2022-11-23T01:48:21.2105639Z Running tests...
2022-11-23T01:48:21.2105940Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2106292Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2106567Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2106823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19826
2022-11-23T01:48:21.2107062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19827
2022-11-23T01:48:21.2107523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2107744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2108165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2108399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2108804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2109018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2109429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2109654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2109926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2110206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2110645Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2111081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2111347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2111609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2111749Z ok (4.348s)
2022-11-23T01:48:21.2111769Z 
2022-11-23T01:48:21.2112067Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2112200Z Ran 1 test in 4.349s
2022-11-23T01:48:21.2112238Z 
2022-11-23T01:48:21.2112351Z OK
2022-11-23T01:48:21.2112370Z 
2022-11-23T01:48:21.2112540Z Generating XML reports...
2022-11-23T01:48:21.2113029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014019.xml
2022-11-23T01:48:21.2113437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2113703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2114120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2114350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2114370Z 
2022-11-23T01:48:21.2114516Z Running tests...
2022-11-23T01:48:21.2114795Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2115149Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2115461Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2115719Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19935
2022-11-23T01:48:21.2115975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19936
2022-11-23T01:48:21.2116388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2116602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2117022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2117234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2117635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2117905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2118328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2118550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2118829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2119114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2119550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2119988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2120238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2120616Z STAGE:2022-11-23 01:40:30 19936:19936 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2120887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2121261Z STAGE:2022-11-23 01:40:30 19935:19935 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2121640Z STAGE:2022-11-23 01:40:30 19935:19935 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2122026Z STAGE:2022-11-23 01:40:30 19935:19935 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2122395Z STAGE:2022-11-23 01:40:30 19936:19936 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2122778Z STAGE:2022-11-23 01:40:30 19936:19936 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2122918Z ok (4.450s)
2022-11-23T01:48:21.2122938Z 
2022-11-23T01:48:21.2123221Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2123373Z Ran 1 test in 4.450s
2022-11-23T01:48:21.2123393Z 
2022-11-23T01:48:21.2123523Z OK
2022-11-23T01:48:21.2123542Z 
2022-11-23T01:48:21.2123703Z Generating XML reports...
2022-11-23T01:48:21.2124188Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014026.xml
2022-11-23T01:48:21.2124651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2124865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2125282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2125496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2125538Z 
2022-11-23T01:48:21.2125872Z Running tests...
2022-11-23T01:48:21.2126193Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2126552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2126858Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2127116Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20048
2022-11-23T01:48:21.2127374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20049
2022-11-23T01:48:21.2127785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2127997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2128398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2128629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2129143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2129367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2129784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2130017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2130300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2130582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2131001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2131437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2131710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2132083Z STAGE:2022-11-23 01:40:37 20049:20049 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2132343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2132711Z STAGE:2022-11-23 01:40:37 20048:20048 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2133084Z STAGE:2022-11-23 01:40:37 20049:20049 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2133469Z STAGE:2022-11-23 01:40:37 20049:20049 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2133840Z STAGE:2022-11-23 01:40:37 20048:20048 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2134207Z STAGE:2022-11-23 01:40:37 20048:20048 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2134353Z ok (4.353s)
2022-11-23T01:48:21.2134377Z 
2022-11-23T01:48:21.2134679Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2134830Z Ran 1 test in 4.353s
2022-11-23T01:48:21.2134849Z 
2022-11-23T01:48:21.2134981Z OK
2022-11-23T01:48:21.2135000Z 
2022-11-23T01:48:21.2135163Z Generating XML reports...
2022-11-23T01:48:21.2135725Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014033.xml
2022-11-23T01:48:21.2136132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2136347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2136748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2136975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2136995Z 
2022-11-23T01:48:21.2137145Z Running tests...
2022-11-23T01:48:21.2137441Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2137794Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2138116Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2138380Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20161
2022-11-23T01:48:21.2138635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20162
2022-11-23T01:48:21.2139027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2139242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2139660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2139934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2140344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2140592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2141014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2141242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2141526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2141790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2142227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2142666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2142933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2143196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2143478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2143757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2144191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2144618Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2144878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:21.2145152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:21.2145576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.2146009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.2146341Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:48:21.2146482Z ok (21.548s)
2022-11-23T01:48:21.2146502Z 
2022-11-23T01:48:21.2146809Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2146963Z Ran 1 test in 21.549s
2022-11-23T01:48:21.2146982Z 
2022-11-23T01:48:21.2147113Z OK
2022-11-23T01:48:21.2147133Z 
2022-11-23T01:48:21.2147276Z Generating XML reports...
2022-11-23T01:48:21.2147771Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014040.xml
2022-11-23T01:48:21.2148186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2148401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2148816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2149047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2149067Z 
2022-11-23T01:48:21.2149214Z Running tests...
2022-11-23T01:48:21.2149513Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2149850Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2150189Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2150491Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20288
2022-11-23T01:48:21.2150757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20289
2022-11-23T01:48:21.2151169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2151386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2151803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2152032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2152436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2152633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2153048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2153275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2153558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2153842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2154281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2154716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2154984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2155260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2155513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2155792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2156224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2156707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2156986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:21.2157262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:21.2157694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.2158128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.2158401Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:48:21.2158652Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:48:21.2158793Z ok (21.796s)
2022-11-23T01:48:21.2158817Z 
2022-11-23T01:48:21.2159122Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2159272Z Ran 1 test in 21.796s
2022-11-23T01:48:21.2159292Z 
2022-11-23T01:48:21.2159420Z OK
2022-11-23T01:48:21.2159440Z 
2022-11-23T01:48:21.2159600Z Generating XML reports...
2022-11-23T01:48:21.2160088Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014104.xml
2022-11-23T01:48:21.2160500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2160743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2161172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2161403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2161423Z 
2022-11-23T01:48:21.2161572Z Running tests...
2022-11-23T01:48:21.2161878Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2162230Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2162546Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2162805Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20415
2022-11-23T01:48:21.2163059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20416
2022-11-23T01:48:21.2163457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2163672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2164091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2164318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2164726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2164939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2165351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2165581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2166143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2166443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2166896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2167330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2167681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2167947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2168143Z skip: Skipped due to small world size. (4.145s)
2022-11-23T01:48:21.2168164Z 
2022-11-23T01:48:21.2168470Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2168621Z Ran 1 test in 4.145s
2022-11-23T01:48:21.2168640Z 
2022-11-23T01:48:21.2168768Z OK (skipped=1)
2022-11-23T01:48:21.2168805Z 
2022-11-23T01:48:21.2168954Z Generating XML reports...
2022-11-23T01:48:21.2169447Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014129.xml
2022-11-23T01:48:21.2169858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2170076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2170494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2170722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2170742Z 
2022-11-23T01:48:21.2170889Z Running tests...
2022-11-23T01:48:21.2171188Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2171524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2171888Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2172154Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20524
2022-11-23T01:48:21.2172408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20525
2022-11-23T01:48:21.2172824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2173038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2173458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2173688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2174070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2174288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2174698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2174922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2175205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2175487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2175921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2176356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2176622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2176875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2177146Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-11-23T01:48:21.2177286Z ok (6.352s)
2022-11-23T01:48:21.2177307Z 
2022-11-23T01:48:21.2177609Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2177813Z Ran 1 test in 6.353s
2022-11-23T01:48:21.2177833Z 
2022-11-23T01:48:21.2177964Z OK
2022-11-23T01:48:21.2177983Z 
2022-11-23T01:48:21.2178144Z Generating XML reports...
2022-11-23T01:48:21.2178633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014135.xml
2022-11-23T01:48:21.2179044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2179241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2179666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2179895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2179915Z 
2022-11-23T01:48:21.2180063Z Running tests...
2022-11-23T01:48:21.2180363Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2180717Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2181038Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2181295Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20633
2022-11-23T01:48:21.2181534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20634
2022-11-23T01:48:21.2181945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2182204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2182630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2182857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2183259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2183476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2183886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2184111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2184378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2184663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2185101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2185531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2185801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2186069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2186347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2186626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2187065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2187485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2187744Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-11-23T01:48:21.2187962Z No ranks successfully processed in monitoredBarrier.
2022-11-23T01:48:21.2188221Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-11-23T01:48:21.2188416Z ok (4.246s)
2022-11-23T01:48:21.2188437Z 
2022-11-23T01:48:21.2188746Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2188899Z Ran 1 test in 4.246s
2022-11-23T01:48:21.2188919Z 
2022-11-23T01:48:21.2189049Z OK
2022-11-23T01:48:21.2189068Z 
2022-11-23T01:48:21.2189212Z Generating XML reports...
2022-11-23T01:48:21.2189704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014144.xml
2022-11-23T01:48:21.2190121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2190333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2190751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2190979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2191003Z 
2022-11-23T01:48:21.2191149Z Running tests...
2022-11-23T01:48:21.2191452Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2191784Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2192102Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2192361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20748
2022-11-23T01:48:21.2192673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20749
2022-11-23T01:48:21.2193090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2193303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2193719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2193952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2194352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2194548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2194958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2195184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2195467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2195747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2196185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2196623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2196891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2197157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2197420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2197704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2198140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2198574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2198896Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:48:21.2199039Z ok (4.520s)
2022-11-23T01:48:21.2199059Z 
2022-11-23T01:48:21.2199362Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2199514Z Ran 1 test in 4.520s
2022-11-23T01:48:21.2199534Z 
2022-11-23T01:48:21.2199663Z OK
2022-11-23T01:48:21.2199682Z 
2022-11-23T01:48:21.2199824Z Generating XML reports...
2022-11-23T01:48:21.2200313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014151.xml
2022-11-23T01:48:21.2200722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2200938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2201358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2201592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2201612Z 
2022-11-23T01:48:21.2201758Z Running tests...
2022-11-23T01:48:21.2202061Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2202397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2202714Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2202972Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20863
2022-11-23T01:48:21.2203278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20864
2022-11-23T01:48:21.2203695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2203909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2204327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2204561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2204960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2205154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2205563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2206010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2206303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2206618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2207060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2207492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2207759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2208001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2208199Z skip: Skipped due to small world size. (4.123s)
2022-11-23T01:48:21.2208219Z 
2022-11-23T01:48:21.2208523Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2208680Z Ran 1 test in 4.123s
2022-11-23T01:48:21.2208700Z 
2022-11-23T01:48:21.2208835Z OK (skipped=1)
2022-11-23T01:48:21.2208854Z 
2022-11-23T01:48:21.2209015Z Generating XML reports...
2022-11-23T01:48:21.2209506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014158.xml
2022-11-23T01:48:21.2210004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2210218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2210618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2210849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2210869Z 
2022-11-23T01:48:21.2211018Z Running tests...
2022-11-23T01:48:21.2211324Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2211677Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2212116Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:48:21.2212137Z 
2022-11-23T01:48:21.2212437Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2212587Z Ran 1 test in 0.003s
2022-11-23T01:48:21.2212608Z 
2022-11-23T01:48:21.2212755Z OK (skipped=1)
2022-11-23T01:48:21.2212774Z 
2022-11-23T01:48:21.2212916Z Generating XML reports...
2022-11-23T01:48:21.2213399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014205.xml
2022-11-23T01:48:21.2213806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2214020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2214490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2214721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2214741Z 
2022-11-23T01:48:21.2214885Z Running tests...
2022-11-23T01:48:21.2215195Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2215528Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2215969Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:48:21.2215989Z 
2022-11-23T01:48:21.2216288Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2216436Z Ran 1 test in 0.003s
2022-11-23T01:48:21.2216456Z 
2022-11-23T01:48:21.2216601Z OK (skipped=1)
2022-11-23T01:48:21.2216620Z 
2022-11-23T01:48:21.2216787Z Generating XML reports...
2022-11-23T01:48:21.2217273Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014207.xml
2022-11-23T01:48:21.2217681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2217900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2218305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2218535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2218555Z 
2022-11-23T01:48:21.2218704Z Running tests...
2022-11-23T01:48:21.2219003Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2219355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2219800Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:48:21.2219821Z 
2022-11-23T01:48:21.2220117Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2220266Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2220286Z 
2022-11-23T01:48:21.2220486Z OK (skipped=1)
2022-11-23T01:48:21.2220506Z 
2022-11-23T01:48:21.2220649Z Generating XML reports...
2022-11-23T01:48:21.2221135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014209.xml
2022-11-23T01:48:21.2221542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2221755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2222171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2222407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2222427Z 
2022-11-23T01:48:21.2222572Z Running tests...
2022-11-23T01:48:21.2222869Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2223221Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2223644Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-11-23T01:48:21.2223665Z 
2022-11-23T01:48:21.2223963Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2224115Z Ran 1 test in 0.003s
2022-11-23T01:48:21.2224134Z 
2022-11-23T01:48:21.2224284Z OK (skipped=1)
2022-11-23T01:48:21.2224303Z 
2022-11-23T01:48:21.2224465Z Generating XML reports...
2022-11-23T01:48:21.2224943Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014212.xml
2022-11-23T01:48:21.2225394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2225614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2226035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2226251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2226271Z 
2022-11-23T01:48:21.2226418Z Running tests...
2022-11-23T01:48:21.2226714Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2227066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2227400Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-11-23T01:48:21.2227420Z 
2022-11-23T01:48:21.2227721Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2227871Z Ran 1 test in 0.003s
2022-11-23T01:48:21.2227891Z 
2022-11-23T01:48:21.2228034Z OK (skipped=1)
2022-11-23T01:48:21.2228053Z 
2022-11-23T01:48:21.2228212Z Generating XML reports...
2022-11-23T01:48:21.2228683Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014214.xml
2022-11-23T01:48:21.2229097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2229311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2229727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2229954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2229974Z 
2022-11-23T01:48:21.2230120Z Running tests...
2022-11-23T01:48:21.2230421Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2230773Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2231043Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:48:21.2231134Z 
2022-11-23T01:48:21.2231418Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2231570Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2231589Z 
2022-11-23T01:48:21.2231733Z OK (skipped=1)
2022-11-23T01:48:21.2231752Z 
2022-11-23T01:48:21.2231909Z Generating XML reports...
2022-11-23T01:48:21.2232394Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014217.xml
2022-11-23T01:48:21.2232797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2233014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2233434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2233645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2233664Z 
2022-11-23T01:48:21.2233810Z Running tests...
2022-11-23T01:48:21.2234112Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2234463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2234771Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:48:21.2234791Z 
2022-11-23T01:48:21.2235093Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2235243Z Ran 1 test in 0.003s
2022-11-23T01:48:21.2235263Z 
2022-11-23T01:48:21.2235407Z OK (skipped=1)
2022-11-23T01:48:21.2235426Z 
2022-11-23T01:48:21.2235631Z Generating XML reports...
2022-11-23T01:48:21.2236107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014219.xml
2022-11-23T01:48:21.2236513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2236731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2237146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2237376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2237396Z 
2022-11-23T01:48:21.2237543Z Running tests...
2022-11-23T01:48:21.2237843Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2238193Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2238527Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:48:21.2238566Z 
2022-11-23T01:48:21.2238846Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2238996Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2239019Z 
2022-11-23T01:48:21.2239162Z OK (skipped=1)
2022-11-23T01:48:21.2239181Z 
2022-11-23T01:48:21.2239340Z Generating XML reports...
2022-11-23T01:48:21.2239820Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014221.xml
2022-11-23T01:48:21.2240230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2240445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2240903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2241120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2241157Z 
2022-11-23T01:48:21.2241287Z Running tests...
2022-11-23T01:48:21.2241584Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2241935Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2242330Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2242592Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21236
2022-11-23T01:48:21.2242847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21237
2022-11-23T01:48:21.2243257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2243469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2243875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2244104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2244505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2244718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2245128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2245354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2245640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2246129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2246627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2247076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2247349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2247615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2247756Z ok (4.167s)
2022-11-23T01:48:21.2247776Z 
2022-11-23T01:48:21.2248081Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2248231Z Ran 1 test in 4.168s
2022-11-23T01:48:21.2248252Z 
2022-11-23T01:48:21.2248381Z OK
2022-11-23T01:48:21.2248400Z 
2022-11-23T01:48:21.2248563Z Generating XML reports...
2022-11-23T01:48:21.2249035Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014224.xml
2022-11-23T01:48:21.2249447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2249662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2250083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2250315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2250335Z 
2022-11-23T01:48:21.2250482Z Running tests...
2022-11-23T01:48:21.2250782Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2251138Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2251472Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2251716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21345
2022-11-23T01:48:21.2251972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21346
2022-11-23T01:48:21.2252385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2252599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2253085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2253314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2253716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2253925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2254314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2254546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2254830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2255111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2255552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2255984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2256250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2256515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2256660Z ok (4.247s)
2022-11-23T01:48:21.2256680Z 
2022-11-23T01:48:21.2257012Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2257173Z Ran 1 test in 4.247s
2022-11-23T01:48:21.2257193Z 
2022-11-23T01:48:21.2257324Z OK
2022-11-23T01:48:21.2257343Z 
2022-11-23T01:48:21.2257505Z Generating XML reports...
2022-11-23T01:48:21.2257994Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014230.xml
2022-11-23T01:48:21.2258407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2258622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2259039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2259251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2259289Z 
2022-11-23T01:48:21.2259419Z Running tests...
2022-11-23T01:48:21.2259723Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2260074Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2260391Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:48:21.2260414Z 
2022-11-23T01:48:21.2260713Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2260863Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2260882Z 
2022-11-23T01:48:21.2261027Z OK (skipped=1)
2022-11-23T01:48:21.2261047Z 
2022-11-23T01:48:21.2261208Z Generating XML reports...
2022-11-23T01:48:21.2261682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014237.xml
2022-11-23T01:48:21.2262093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2262312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2262729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2262955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2262974Z 
2022-11-23T01:48:21.2263118Z Running tests...
2022-11-23T01:48:21.2263473Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2263825Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2264162Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:48:21.2264182Z 
2022-11-23T01:48:21.2264462Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2264613Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2264634Z 
2022-11-23T01:48:21.2264781Z OK (skipped=1)
2022-11-23T01:48:21.2264801Z 
2022-11-23T01:48:21.2265021Z Generating XML reports...
2022-11-23T01:48:21.2265900Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014240.xml
2022-11-23T01:48:21.2266352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2266571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2266987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2267218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2267238Z 
2022-11-23T01:48:21.2267369Z Running tests...
2022-11-23T01:48:21.2267671Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2268022Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2268418Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2269220Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.629s)
2022-11-23T01:48:21.2269245Z 
2022-11-23T01:48:21.2269547Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2269699Z Ran 1 test in 1.629s
2022-11-23T01:48:21.2269719Z 
2022-11-23T01:48:21.2269867Z OK (skipped=1)
2022-11-23T01:48:21.2269887Z 
2022-11-23T01:48:21.2270048Z Generating XML reports...
2022-11-23T01:48:21.2270516Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014242.xml
2022-11-23T01:48:21.2270933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2271149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2271562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2271794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2271813Z 
2022-11-23T01:48:21.2271960Z Running tests...
2022-11-23T01:48:21.2272260Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2272612Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2272929Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2273169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21554
2022-11-23T01:48:21.2273429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21555
2022-11-23T01:48:21.2273840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2274053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2274467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2274754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2275159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2275370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2275767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2275999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2276285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2276566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2277007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2277447Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2277713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2277981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2278277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv29ea_kz
2022-11-23T01:48:21.2278613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv29ea_kz/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2278915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemwt1k3e
2022-11-23T01:48:21.2279221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemwt1k3e/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2279364Z ok (5.649s)
2022-11-23T01:48:21.2279387Z 
2022-11-23T01:48:21.2279695Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2279845Z Ran 1 test in 5.649s
2022-11-23T01:48:21.2279865Z 
2022-11-23T01:48:21.2279993Z OK
2022-11-23T01:48:21.2280012Z 
2022-11-23T01:48:21.2280172Z Generating XML reports...
2022-11-23T01:48:21.2280659Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014246.xml
2022-11-23T01:48:21.2281053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2281272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2281693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2281921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2281941Z 
2022-11-23T01:48:21.2282092Z Running tests...
2022-11-23T01:48:21.2282389Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2282739Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2283044Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2283286Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21669
2022-11-23T01:48:21.2283541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21670
2022-11-23T01:48:21.2283949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2284162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2284576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2284802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2285255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2285465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2286125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2286343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2286629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2286915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2287361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2287793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2288063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2288328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2288467Z ok (5.547s)
2022-11-23T01:48:21.2288487Z 
2022-11-23T01:48:21.2288788Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2288922Z Ran 1 test in 5.547s
2022-11-23T01:48:21.2288942Z 
2022-11-23T01:48:21.2289072Z OK
2022-11-23T01:48:21.2289091Z 
2022-11-23T01:48:21.2289253Z Generating XML reports...
2022-11-23T01:48:21.2289817Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014254.xml
2022-11-23T01:48:21.2290234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2290450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2290867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2291097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2291117Z 
2022-11-23T01:48:21.2291246Z Running tests...
2022-11-23T01:48:21.2291545Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2291896Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2292224Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2292482Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21781
2022-11-23T01:48:21.2292736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21782
2022-11-23T01:48:21.2293142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2293359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2293777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2293988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2294392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2294604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2295017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2295243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2295526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2295924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2296369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2296786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2297056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2297323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2297461Z ok (5.550s)
2022-11-23T01:48:21.2297482Z 
2022-11-23T01:48:21.2297782Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2297933Z Ran 1 test in 5.550s
2022-11-23T01:48:21.2297953Z 
2022-11-23T01:48:21.2298082Z OK
2022-11-23T01:48:21.2298101Z 
2022-11-23T01:48:21.2298264Z Generating XML reports...
2022-11-23T01:48:21.2298748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014303.xml
2022-11-23T01:48:21.2299140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2299351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2299768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2300041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2300064Z 
2022-11-23T01:48:21.2300216Z Running tests...
2022-11-23T01:48:21.2300518Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2300870Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2301184Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2301975Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.647s)
2022-11-23T01:48:21.2301996Z 
2022-11-23T01:48:21.2302291Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2302427Z Ran 1 test in 1.647s
2022-11-23T01:48:21.2302446Z 
2022-11-23T01:48:21.2302598Z OK (skipped=1)
2022-11-23T01:48:21.2302618Z 
2022-11-23T01:48:21.2302778Z Generating XML reports...
2022-11-23T01:48:21.2303266Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014311.xml
2022-11-23T01:48:21.2303675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2303893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2304313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2304542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2304562Z 
2022-11-23T01:48:21.2304690Z Running tests...
2022-11-23T01:48:21.2304991Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2305347Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2305680Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2306461Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.614s)
2022-11-23T01:48:21.2306528Z 
2022-11-23T01:48:21.2306830Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2306981Z Ran 1 test in 1.614s
2022-11-23T01:48:21.2307001Z 
2022-11-23T01:48:21.2307146Z OK (skipped=1)
2022-11-23T01:48:21.2307165Z 
2022-11-23T01:48:21.2307326Z Generating XML reports...
2022-11-23T01:48:21.2307813Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014315.xml
2022-11-23T01:48:21.2308208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2308424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2308846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2309076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2309095Z 
2022-11-23T01:48:21.2309242Z Running tests...
2022-11-23T01:48:21.2309540Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2309889Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2310234Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2310525Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21961
2022-11-23T01:48:21.2310788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21962
2022-11-23T01:48:21.2311201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2311413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2311836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2312063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2312469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2312680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2313094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2313311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2313598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2313881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2314326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2314759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2315024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2315290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2315478Z skip: Need at least 4 CUDA devices (4.210s)
2022-11-23T01:48:21.2315499Z 
2022-11-23T01:48:21.2315804Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2315939Z Ran 1 test in 4.210s
2022-11-23T01:48:21.2315959Z 
2022-11-23T01:48:21.2316105Z OK (skipped=1)
2022-11-23T01:48:21.2316123Z 
2022-11-23T01:48:21.2316284Z Generating XML reports...
2022-11-23T01:48:21.2316775Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014319.xml
2022-11-23T01:48:21.2317240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2317451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2317865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2318095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2318115Z 
2022-11-23T01:48:21.2318262Z Running tests...
2022-11-23T01:48:21.2318552Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2318905Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2319269Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2319533Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22070
2022-11-23T01:48:21.2319785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22071
2022-11-23T01:48:21.2320195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2320411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2320827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2321088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2321497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2321709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2322123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2322353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2322637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2322917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2323355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2323792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2324043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2324312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2324502Z skip: Need at least 4 CUDA devices (4.118s)
2022-11-23T01:48:21.2324523Z 
2022-11-23T01:48:21.2324828Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2324979Z Ran 1 test in 4.119s
2022-11-23T01:48:21.2324999Z 
2022-11-23T01:48:21.2325146Z OK (skipped=1)
2022-11-23T01:48:21.2325165Z 
2022-11-23T01:48:21.2325328Z Generating XML reports...
2022-11-23T01:48:21.2326034Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014326.xml
2022-11-23T01:48:21.2326443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2326659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2327075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2327304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2327398Z 
2022-11-23T01:48:21.2327552Z Running tests...
2022-11-23T01:48:21.2327855Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2328205Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2328526Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2329320Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.648s)
2022-11-23T01:48:21.2329341Z 
2022-11-23T01:48:21.2329638Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2329771Z Ran 1 test in 1.648s
2022-11-23T01:48:21.2329790Z 
2022-11-23T01:48:21.2329942Z OK (skipped=1)
2022-11-23T01:48:21.2329962Z 
2022-11-23T01:48:21.2330121Z Generating XML reports...
2022-11-23T01:48:21.2330607Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014332.xml
2022-11-23T01:48:21.2331012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2331225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2331642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2331929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2331951Z 
2022-11-23T01:48:21.2332099Z Running tests...
2022-11-23T01:48:21.2332385Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2332736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2333040Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2333294Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22213
2022-11-23T01:48:21.2333549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22214
2022-11-23T01:48:21.2333960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2334172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2334590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2334801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2335202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2335418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2335827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2336052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2336333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2336615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2337055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2337494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2337747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2338067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2338343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2338619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2339052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2339479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2339857Z STAGE:2022-11-23 01:43:40 22213:22213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2340218Z STAGE:2022-11-23 01:43:40 22214:22214 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2340626Z STAGE:2022-11-23 01:43:40 22214:22214 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2340983Z STAGE:2022-11-23 01:43:40 22213:22213 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2341368Z STAGE:2022-11-23 01:43:40 22214:22214 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2341752Z STAGE:2022-11-23 01:43:40 22213:22213 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2342111Z STAGE:2022-11-23 01:43:40 22214:22214 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2342525Z STAGE:2022-11-23 01:43:40 22213:22213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2342901Z STAGE:2022-11-23 01:43:40 22213:22213 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2343270Z STAGE:2022-11-23 01:43:40 22214:22214 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2343650Z STAGE:2022-11-23 01:43:40 22213:22213 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2344035Z STAGE:2022-11-23 01:43:40 22214:22214 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2344158Z ok (4.439s)
2022-11-23T01:48:21.2344179Z 
2022-11-23T01:48:21.2344480Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2344626Z Ran 1 test in 4.439s
2022-11-23T01:48:21.2344646Z 
2022-11-23T01:48:21.2344777Z OK
2022-11-23T01:48:21.2344795Z 
2022-11-23T01:48:21.2344957Z Generating XML reports...
2022-11-23T01:48:21.2345448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014337.xml
2022-11-23T01:48:21.2345861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2346072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2346473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2346711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2346731Z 
2022-11-23T01:48:21.2346879Z Running tests...
2022-11-23T01:48:21.2347177Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2347531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2347832Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2348092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22332
2022-11-23T01:48:21.2348348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22333
2022-11-23T01:48:21.2348758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2349007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2349426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2349657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2350059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2350268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2350682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2350907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2351186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2351450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2351893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2352330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2352597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2352860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2353184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2353470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2353909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2354340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2354719Z STAGE:2022-11-23 01:43:47 22333:22333 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2355061Z STAGE:2022-11-23 01:43:47 22332:22332 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2355433Z STAGE:2022-11-23 01:43:47 22333:22333 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2355796Z STAGE:2022-11-23 01:43:47 22332:22332 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2356404Z STAGE:2022-11-23 01:43:47 22333:22333 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:43:47 22332:22332 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2356425Z 
2022-11-23T01:48:21.2356791Z STAGE:2022-11-23 01:43:47 22333:22333 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2357158Z STAGE:2022-11-23 01:43:47 22332:22332 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2357737Z STAGE:2022-11-23 01:43:47 22332:22332 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:43:47 22333:22333 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2357759Z 
2022-11-23T01:48:21.2358354Z STAGE:2022-11-23 01:43:47 22332:22332 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:43:47 22333:22333 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2358375Z 
2022-11-23T01:48:21.2358519Z ok (4.220s)
2022-11-23T01:48:21.2358540Z 
2022-11-23T01:48:21.2358840Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2358991Z Ran 1 test in 4.220s
2022-11-23T01:48:21.2359011Z 
2022-11-23T01:48:21.2359121Z OK
2022-11-23T01:48:21.2359140Z 
2022-11-23T01:48:21.2359302Z Generating XML reports...
2022-11-23T01:48:21.2359850Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014343.xml
2022-11-23T01:48:21.2360263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2360477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2360896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2361129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2361153Z 
2022-11-23T01:48:21.2361300Z Running tests...
2022-11-23T01:48:21.2361583Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2361935Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2362245Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2362505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22451
2022-11-23T01:48:21.2362759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22452
2022-11-23T01:48:21.2363167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2363378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2363799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2364077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2364473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2364682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2365094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2365321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2365605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2366097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2366545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2366986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2367257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2367507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2367788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2368066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2368500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2368925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2369300Z STAGE:2022-11-23 01:43:54 22451:22451 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2369662Z STAGE:2022-11-23 01:43:54 22452:22452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2370033Z STAGE:2022-11-23 01:43:54 22452:22452 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2370395Z STAGE:2022-11-23 01:43:54 22451:22451 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2370846Z STAGE:2022-11-23 01:43:54 22452:22452 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2371229Z STAGE:2022-11-23 01:43:54 22451:22451 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2371593Z STAGE:2022-11-23 01:43:54 22452:22452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2371947Z STAGE:2022-11-23 01:43:54 22451:22451 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2372319Z STAGE:2022-11-23 01:43:54 22451:22451 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2372687Z STAGE:2022-11-23 01:43:54 22452:22452 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2373065Z STAGE:2022-11-23 01:43:54 22451:22451 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2373452Z STAGE:2022-11-23 01:43:54 22452:22452 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2373576Z ok (4.374s)
2022-11-23T01:48:21.2373615Z 
2022-11-23T01:48:21.2373900Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2374056Z Ran 1 test in 4.374s
2022-11-23T01:48:21.2374075Z 
2022-11-23T01:48:21.2374207Z OK
2022-11-23T01:48:21.2374226Z 
2022-11-23T01:48:21.2374387Z Generating XML reports...
2022-11-23T01:48:21.2374879Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014350.xml
2022-11-23T01:48:21.2375354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2375577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2376002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2376220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2376240Z 
2022-11-23T01:48:21.2376390Z Running tests...
2022-11-23T01:48:21.2376693Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2377046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2377344Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2377602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22570
2022-11-23T01:48:21.2377859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22571
2022-11-23T01:48:21.2378268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2378466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2378893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2379123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2379524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2379734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2380143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2380371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2380655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2380938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2381360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2381854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2382124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2382390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2382665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2382945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2383382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2383808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2384183Z STAGE:2022-11-23 01:44:01 22570:22570 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2384525Z STAGE:2022-11-23 01:44:01 22571:22571 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2385109Z STAGE:2022-11-23 01:44:01 22570:22570 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:44:01 22571:22571 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2385130Z 
2022-11-23T01:48:21.2385780Z STAGE:2022-11-23 01:44:01 22570:22570 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:44:01 22571:22571 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2385803Z 
2022-11-23T01:48:21.2386178Z STAGE:2022-11-23 01:44:01 22571:22571 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2386535Z STAGE:2022-11-23 01:44:01 22570:22570 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2386908Z STAGE:2022-11-23 01:44:01 22571:22571 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2387270Z STAGE:2022-11-23 01:44:01 22570:22570 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2387650Z STAGE:2022-11-23 01:44:01 22571:22571 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2388033Z STAGE:2022-11-23 01:44:01 22570:22570 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2388174Z ok (4.310s)
2022-11-23T01:48:21.2388194Z 
2022-11-23T01:48:21.2388503Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2388639Z Ran 1 test in 4.311s
2022-11-23T01:48:21.2388659Z 
2022-11-23T01:48:21.2388791Z OK
2022-11-23T01:48:21.2388810Z 
2022-11-23T01:48:21.2388977Z Generating XML reports...
2022-11-23T01:48:21.2389465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014357.xml
2022-11-23T01:48:21.2389877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2390089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2390507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2390736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2390756Z 
2022-11-23T01:48:21.2390885Z Running tests...
2022-11-23T01:48:21.2391192Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2391546Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2391839Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2392151Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22689
2022-11-23T01:48:21.2392406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22690
2022-11-23T01:48:21.2392818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2393031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2393434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2393664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2394077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2394286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2394701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2394934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2395217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2395654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2395934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2396399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2396676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2396944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2397138Z skip: Skipped due to small world size. (4.348s)
2022-11-23T01:48:21.2397162Z 
2022-11-23T01:48:21.2397467Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2397617Z Ran 1 test in 4.349s
2022-11-23T01:48:21.2397637Z 
2022-11-23T01:48:21.2397782Z OK (skipped=1)
2022-11-23T01:48:21.2397802Z 
2022-11-23T01:48:21.2397963Z Generating XML reports...
2022-11-23T01:48:21.2398450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014404.xml
2022-11-23T01:48:21.2398842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2399063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2399481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2399709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2399732Z 
2022-11-23T01:48:21.2399879Z Running tests...
2022-11-23T01:48:21.2400181Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2400529Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2400822Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2401064Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22798
2022-11-23T01:48:21.2401319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22799
2022-11-23T01:48:21.2401730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2401943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2402359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2402639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2403039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2403253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2403664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2403874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2404157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2404447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2404886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2405317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2405590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2406064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2406268Z skip: Skipped due to small world size. (4.263s)
2022-11-23T01:48:21.2406289Z 
2022-11-23T01:48:21.2406594Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2406727Z Ran 1 test in 4.264s
2022-11-23T01:48:21.2406747Z 
2022-11-23T01:48:21.2406892Z OK (skipped=1)
2022-11-23T01:48:21.2406983Z 
2022-11-23T01:48:21.2407156Z Generating XML reports...
2022-11-23T01:48:21.2407655Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014411.xml
2022-11-23T01:48:21.2408063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2408278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2408694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2408922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2408942Z 
2022-11-23T01:48:21.2409087Z Running tests...
2022-11-23T01:48:21.2409371Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2409723Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2410030Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2410288Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22907
2022-11-23T01:48:21.2410543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22908
2022-11-23T01:48:21.2410959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2411171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2411584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2411797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2412199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2412410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2412820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2413044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2413326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2413677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2414120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2414556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2414810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2415077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2415274Z skip: Skipped due to small world size. (4.239s)
2022-11-23T01:48:21.2415295Z 
2022-11-23T01:48:21.2415603Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2415754Z Ran 1 test in 4.239s
2022-11-23T01:48:21.2415779Z 
2022-11-23T01:48:21.2415926Z OK (skipped=1)
2022-11-23T01:48:21.2415947Z 
2022-11-23T01:48:21.2416110Z Generating XML reports...
2022-11-23T01:48:21.2416595Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014418.xml
2022-11-23T01:48:21.2417006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2417204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2417682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2417921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2417941Z 
2022-11-23T01:48:21.2418087Z Running tests...
2022-11-23T01:48:21.2418389Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2418740Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2419039Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2419295Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23016
2022-11-23T01:48:21.2419532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23017
2022-11-23T01:48:21.2419944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2420158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2420580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2420808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2421214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2421430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2421848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2422074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2422342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2422625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2423067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2423505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2423770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2424092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2424289Z skip: Skipped due to small world size. (4.209s)
2022-11-23T01:48:21.2424308Z 
2022-11-23T01:48:21.2424610Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2424744Z Ran 1 test in 4.209s
2022-11-23T01:48:21.2424783Z 
2022-11-23T01:48:21.2424911Z OK (skipped=1)
2022-11-23T01:48:21.2424931Z 
2022-11-23T01:48:21.2425090Z Generating XML reports...
2022-11-23T01:48:21.2425584Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014424.xml
2022-11-23T01:48:21.2425999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2426212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2426626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2426859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2426879Z 
2022-11-23T01:48:21.2427029Z Running tests...
2022-11-23T01:48:21.2427312Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2427666Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2427951Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2428253Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23125
2022-11-23T01:48:21.2428515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23126
2022-11-23T01:48:21.2428923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2429134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2429556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2429767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2430166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2430378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2430789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2431020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2431301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2431578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2432022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2432451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2432699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2433073Z STAGE:2022-11-23 01:44:35 23126:23126 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2433336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2433711Z STAGE:2022-11-23 01:44:35 23125:23125 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2434296Z STAGE:2022-11-23 01:44:35 23126:23126 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:44:35 23125:23125 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2434366Z 
2022-11-23T01:48:21.2434980Z STAGE:2022-11-23 01:44:35 23125:23125 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:44:35 23126:23126 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2435001Z 
2022-11-23T01:48:21.2435365Z STAGE:2022-11-23 01:44:35 23126:23126 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2435722Z STAGE:2022-11-23 01:44:35 23125:23125 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2436308Z STAGE:2022-11-23 01:44:35 23125:23125 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:44:35 23126:23126 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2436329Z 
2022-11-23T01:48:21.2436929Z STAGE:2022-11-23 01:44:35 23125:23125 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:44:35 23126:23126 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2436954Z 
2022-11-23T01:48:21.2437094Z ok (4.339s)
2022-11-23T01:48:21.2437114Z 
2022-11-23T01:48:21.2437414Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2437547Z Ran 1 test in 4.340s
2022-11-23T01:48:21.2437566Z 
2022-11-23T01:48:21.2437695Z OK
2022-11-23T01:48:21.2437714Z 
2022-11-23T01:48:21.2437871Z Generating XML reports...
2022-11-23T01:48:21.2438361Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014431.xml
2022-11-23T01:48:21.2438814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2439036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2439454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2439689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2439709Z 
2022-11-23T01:48:21.2439857Z Running tests...
2022-11-23T01:48:21.2440138Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2440489Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2440819Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2441077Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23238
2022-11-23T01:48:21.2441336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23239
2022-11-23T01:48:21.2441749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2441961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2442381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2442597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2443000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2443209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2443616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2443840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2444123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2444402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2444841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2445334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2445585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2446176Z STAGE:2022-11-23 01:44:42 23239:23239 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2446442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2446820Z STAGE:2022-11-23 01:44:42 23238:23238 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2447408Z STAGE:2022-11-23 01:44:42 23239:23239 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:44:42 23238:23238 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2447430Z 
2022-11-23T01:48:21.2448027Z STAGE:2022-11-23 01:44:42 23239:23239 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:44:42 23238:23238 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2448052Z 
2022-11-23T01:48:21.2448416Z STAGE:2022-11-23 01:44:42 23239:23239 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2448773Z STAGE:2022-11-23 01:44:42 23238:23238 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2449139Z STAGE:2022-11-23 01:44:42 23238:23238 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2449578Z STAGE:2022-11-23 01:44:42 23239:23239 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2449955Z STAGE:2022-11-23 01:44:42 23238:23238 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2450338Z STAGE:2022-11-23 01:44:42 23239:23239 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2450483Z ok (4.362s)
2022-11-23T01:48:21.2450503Z 
2022-11-23T01:48:21.2450803Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2450952Z Ran 1 test in 4.363s
2022-11-23T01:48:21.2450972Z 
2022-11-23T01:48:21.2451101Z OK
2022-11-23T01:48:21.2451120Z 
2022-11-23T01:48:21.2451283Z Generating XML reports...
2022-11-23T01:48:21.2451773Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014438.xml
2022-11-23T01:48:21.2452182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2452383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2452801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2453029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2453052Z 
2022-11-23T01:48:21.2453198Z Running tests...
2022-11-23T01:48:21.2453498Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2453847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2454162Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-11-23T01:48:21.2454182Z 
2022-11-23T01:48:21.2454475Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2454626Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2454645Z 
2022-11-23T01:48:21.2454777Z OK (skipped=1)
2022-11-23T01:48:21.2454801Z 
2022-11-23T01:48:21.2454965Z Generating XML reports...
2022-11-23T01:48:21.2455456Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014445.xml
2022-11-23T01:48:21.2455860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2456142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2456568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2456799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2456819Z 
2022-11-23T01:48:21.2456963Z Running tests...
2022-11-23T01:48:21.2457246Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2457599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2457895Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2458155Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23384
2022-11-23T01:48:21.2458409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23385
2022-11-23T01:48:21.2458824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2459036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2459448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2459673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2460060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2460350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2460771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2460996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2461280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2461564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2462000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2462436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2462704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2463064Z STAGE:2022-11-23 01:44:51 23385:23385 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2463325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2463689Z STAGE:2022-11-23 01:44:51 23384:23384 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2464275Z STAGE:2022-11-23 01:44:51 23385:23385 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:44:51 23384:23384 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2464296Z 
2022-11-23T01:48:21.2464899Z STAGE:2022-11-23 01:44:51 23384:23384 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:44:51 23385:23385 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2464920Z 
2022-11-23T01:48:21.2465281Z STAGE:2022-11-23 01:44:51 23385:23385 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2465644Z STAGE:2022-11-23 01:44:51 23384:23384 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2466011Z STAGE:2022-11-23 01:44:51 23385:23385 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2466372Z STAGE:2022-11-23 01:44:51 23384:23384 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2466814Z STAGE:2022-11-23 01:44:51 23385:23385 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2467180Z STAGE:2022-11-23 01:44:51 23384:23384 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2467321Z ok (4.362s)
2022-11-23T01:48:21.2467341Z 
2022-11-23T01:48:21.2467642Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2467791Z Ran 1 test in 4.363s
2022-11-23T01:48:21.2467810Z 
2022-11-23T01:48:21.2467935Z OK
2022-11-23T01:48:21.2467955Z 
2022-11-23T01:48:21.2468113Z Generating XML reports...
2022-11-23T01:48:21.2468604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014447.xml
2022-11-23T01:48:21.2469013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2469229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2469634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2469867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2469887Z 
2022-11-23T01:48:21.2470032Z Running tests...
2022-11-23T01:48:21.2470331Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2470681Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2471055Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2022-11-23T01:48:21.2471078Z 
2022-11-23T01:48:21.2471381Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2471533Z Ran 1 test in 0.003s
2022-11-23T01:48:21.2471552Z 
2022-11-23T01:48:21.2471681Z OK (skipped=1)
2022-11-23T01:48:21.2471716Z 
2022-11-23T01:48:21.2471865Z Generating XML reports...
2022-11-23T01:48:21.2472349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014454.xml
2022-11-23T01:48:21.2472757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2472969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2473383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2473613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2473634Z 
2022-11-23T01:48:21.2473780Z Running tests...
2022-11-23T01:48:21.2474073Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2474407Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2474726Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-11-23T01:48:21.2474746Z 
2022-11-23T01:48:21.2475039Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2475187Z Ran 1 test in 0.003s
2022-11-23T01:48:21.2475207Z 
2022-11-23T01:48:21.2475351Z OK (skipped=1)
2022-11-23T01:48:21.2475370Z 
2022-11-23T01:48:21.2475529Z Generating XML reports...
2022-11-23T01:48:21.2476019Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014457.xml
2022-11-23T01:48:21.2476433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2476646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2477048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2477324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2477344Z 
2022-11-23T01:48:21.2477492Z Running tests...
2022-11-23T01:48:21.2477794Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2478144Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2478428Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2478684Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23563
2022-11-23T01:48:21.2478942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23564
2022-11-23T01:48:21.2479337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2479548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2479964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2480194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2480602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2480813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2481226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2481450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2481776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2482048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2482490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2482931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2483201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2483464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2483832Z STAGE:2022-11-23 01:45:03 23563:23563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2484196Z STAGE:2022-11-23 01:45:03 23564:23564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2484781Z STAGE:2022-11-23 01:45:03 23564:23564 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:45:03 23563:23563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2484801Z 
2022-11-23T01:48:21.2485404Z STAGE:2022-11-23 01:45:03 23563:23563 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:45:03 23564:23564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2485428Z 
2022-11-23T01:48:21.2485995Z STAGE:2022-11-23 01:45:03 23564:23564 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2486362Z STAGE:2022-11-23 01:45:03 23563:23563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2486731Z STAGE:2022-11-23 01:45:03 23563:23563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2487098Z STAGE:2022-11-23 01:45:03 23564:23564 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2487481Z STAGE:2022-11-23 01:45:03 23563:23563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2487849Z STAGE:2022-11-23 01:45:03 23564:23564 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2488067Z ok (4.233s)
2022-11-23T01:48:21.2488088Z 
2022-11-23T01:48:21.2488386Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2488534Z Ran 1 test in 4.234s
2022-11-23T01:48:21.2488554Z 
2022-11-23T01:48:21.2488682Z OK
2022-11-23T01:48:21.2488701Z 
2022-11-23T01:48:21.2488862Z Generating XML reports...
2022-11-23T01:48:21.2489353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014459.xml
2022-11-23T01:48:21.2489763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2489964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2490386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2490611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2490635Z 
2022-11-23T01:48:21.2490773Z Running tests...
2022-11-23T01:48:21.2491072Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2491422Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2491716Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-11-23T01:48:21.2491736Z 
2022-11-23T01:48:21.2492031Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2492181Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2492201Z 
2022-11-23T01:48:21.2492331Z OK (skipped=1)
2022-11-23T01:48:21.2492350Z 
2022-11-23T01:48:21.2492583Z Generating XML reports...
2022-11-23T01:48:21.2493085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014506.xml
2022-11-23T01:48:21.2493491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2493708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2494124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2494351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2494371Z 
2022-11-23T01:48:21.2494517Z Running tests...
2022-11-23T01:48:21.2494801Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2495152Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2495453Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-11-23T01:48:21.2495473Z 
2022-11-23T01:48:21.2495772Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2495919Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2495939Z 
2022-11-23T01:48:21.2496080Z OK (skipped=1)
2022-11-23T01:48:21.2496099Z 
2022-11-23T01:48:21.2496256Z Generating XML reports...
2022-11-23T01:48:21.2496741Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014508.xml
2022-11-23T01:48:21.2497144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2497341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2497754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2497985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2498004Z 
2022-11-23T01:48:21.2498149Z Running tests...
2022-11-23T01:48:21.2498446Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2498794Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2499139Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2499397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23742
2022-11-23T01:48:21.2499652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23743
2022-11-23T01:48:21.2500049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2500260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2500681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2500911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2501313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2501524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2501933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2502158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2502425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2502704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2503182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2503623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2503885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2504260Z STAGE:2022-11-23 01:45:15 23743:23743 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2504525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2504889Z STAGE:2022-11-23 01:45:15 23742:23742 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2505470Z STAGE:2022-11-23 01:45:15 23743:23743 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:45:15 23742:23742 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2505490Z 
2022-11-23T01:48:21.2506102Z STAGE:2022-11-23 01:45:15 23743:23743 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:45:15 23742:23742 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2506123Z 
2022-11-23T01:48:21.2506531Z STAGE:2022-11-23 01:45:15 23743:23743 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2506884Z STAGE:2022-11-23 01:45:15 23742:23742 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2507251Z STAGE:2022-11-23 01:45:15 23742:23742 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2507617Z STAGE:2022-11-23 01:45:15 23743:23743 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2507999Z STAGE:2022-11-23 01:45:15 23742:23742 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2508377Z STAGE:2022-11-23 01:45:15 23743:23743 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2508521Z ok (4.372s)
2022-11-23T01:48:21.2508541Z 
2022-11-23T01:48:21.2508840Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2508988Z Ran 1 test in 4.372s
2022-11-23T01:48:21.2509008Z 
2022-11-23T01:48:21.2509120Z OK
2022-11-23T01:48:21.2509154Z 
2022-11-23T01:48:21.2509298Z Generating XML reports...
2022-11-23T01:48:21.2509846Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014511.xml
2022-11-23T01:48:21.2510252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2510462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2510876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2511103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2511127Z 
2022-11-23T01:48:21.2511271Z Running tests...
2022-11-23T01:48:21.2511571Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2511907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2512188Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2512447Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23855
2022-11-23T01:48:21.2512699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23856
2022-11-23T01:48:21.2513105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2513315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2513727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2514000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2514391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2514601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2515013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2515233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2515514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2515791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2516227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2516665Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2516931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2517180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2517556Z STAGE:2022-11-23 01:45:22 23856:23856 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2517912Z STAGE:2022-11-23 01:45:22 23855:23855 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2518280Z STAGE:2022-11-23 01:45:22 23856:23856 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2518637Z STAGE:2022-11-23 01:45:22 23855:23855 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2519016Z STAGE:2022-11-23 01:45:22 23856:23856 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2519399Z STAGE:2022-11-23 01:45:22 23855:23855 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2519760Z STAGE:2022-11-23 01:45:22 23855:23855 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2520117Z STAGE:2022-11-23 01:45:22 23856:23856 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2520522Z STAGE:2022-11-23 01:45:22 23856:23856 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2520886Z STAGE:2022-11-23 01:45:22 23855:23855 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2521266Z STAGE:2022-11-23 01:45:22 23856:23856 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2521640Z STAGE:2022-11-23 01:45:22 23855:23855 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2521779Z ok (4.258s)
2022-11-23T01:48:21.2521799Z 
2022-11-23T01:48:21.2522102Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2522252Z Ran 1 test in 4.258s
2022-11-23T01:48:21.2522271Z 
2022-11-23T01:48:21.2522397Z OK
2022-11-23T01:48:21.2522416Z 
2022-11-23T01:48:21.2522574Z Generating XML reports...
2022-11-23T01:48:21.2523047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014518.xml
2022-11-23T01:48:21.2523460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2523671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2524085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2524312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2524332Z 
2022-11-23T01:48:21.2524475Z Running tests...
2022-11-23T01:48:21.2524816Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2525176Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2525452Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2525917Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23968
2022-11-23T01:48:21.2526187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23969
2022-11-23T01:48:21.2526598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2526804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2527213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2527437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2527840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2528054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2528450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2528678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2528958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2529234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2529668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2530101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2530372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2530638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2530761Z ok (4.153s)
2022-11-23T01:48:21.2530796Z 
2022-11-23T01:48:21.2531166Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2531314Z Ran 1 test in 4.153s
2022-11-23T01:48:21.2531334Z 
2022-11-23T01:48:21.2531459Z OK
2022-11-23T01:48:21.2531478Z 
2022-11-23T01:48:21.2531635Z Generating XML reports...
2022-11-23T01:48:21.2532121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014524.xml
2022-11-23T01:48:21.2532527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2532735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2533155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2533367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2533403Z 
2022-11-23T01:48:21.2533535Z Running tests...
2022-11-23T01:48:21.2533838Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2534185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2534474Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2534722Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24077
2022-11-23T01:48:21.2534971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24078
2022-11-23T01:48:21.2535373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2535629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2536054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2536275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2536671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2536876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2537279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2537496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2537773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2538051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2538473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2538902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2539172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2539544Z STAGE:2022-11-23 01:45:35 24078:24078 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2539804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2540163Z STAGE:2022-11-23 01:45:35 24077:24077 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2540532Z STAGE:2022-11-23 01:45:35 24077:24077 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2540939Z STAGE:2022-11-23 01:45:35 24078:24078 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2541321Z STAGE:2022-11-23 01:45:35 24077:24077 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2541690Z STAGE:2022-11-23 01:45:35 24078:24078 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2542115Z STAGE:2022-11-23 01:45:35 24077:24077 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2542471Z STAGE:2022-11-23 01:45:35 24078:24078 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2543047Z STAGE:2022-11-23 01:45:35 24078:24078 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:45:35 24077:24077 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2543068Z 
2022-11-23T01:48:21.2543447Z STAGE:2022-11-23 01:45:35 24077:24077 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2543830Z STAGE:2022-11-23 01:45:35 24078:24078 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2543964Z ok (4.347s)
2022-11-23T01:48:21.2543984Z 
2022-11-23T01:48:21.2544282Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2544435Z Ran 1 test in 4.347s
2022-11-23T01:48:21.2544454Z 
2022-11-23T01:48:21.2544566Z OK
2022-11-23T01:48:21.2544601Z 
2022-11-23T01:48:21.2544746Z Generating XML reports...
2022-11-23T01:48:21.2545234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014531.xml
2022-11-23T01:48:21.2545639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2545853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2546320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2546554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2546574Z 
2022-11-23T01:48:21.2546720Z Running tests...
2022-11-23T01:48:21.2547020Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2547356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2547655Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T01:48:21.2547676Z 
2022-11-23T01:48:21.2547971Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2548116Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2548137Z 
2022-11-23T01:48:21.2548278Z OK (skipped=1)
2022-11-23T01:48:21.2548297Z 
2022-11-23T01:48:21.2548451Z Generating XML reports...
2022-11-23T01:48:21.2548939Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014538.xml
2022-11-23T01:48:21.2549347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2549558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2549960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2550192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2550212Z 
2022-11-23T01:48:21.2550356Z Running tests...
2022-11-23T01:48:21.2550654Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2551001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2551301Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T01:48:21.2551321Z 
2022-11-23T01:48:21.2551623Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2551771Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2551791Z 
2022-11-23T01:48:21.2551936Z OK (skipped=1)
2022-11-23T01:48:21.2551955Z 
2022-11-23T01:48:21.2552098Z Generating XML reports...
2022-11-23T01:48:21.2552582Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014540.xml
2022-11-23T01:48:21.2553045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2553256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2553673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2553898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2553918Z 
2022-11-23T01:48:21.2554061Z Running tests...
2022-11-23T01:48:21.2554363Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2554700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2554994Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2555255Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24256
2022-11-23T01:48:21.2555508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24257
2022-11-23T01:48:21.2555916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2556126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2556543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2556770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2557215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2557418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2557833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2558059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2558337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2558616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2559051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2559480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2559750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2560000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2560279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2560563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2560999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2561429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2561798Z STAGE:2022-11-23 01:45:47 24257:24257 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2562159Z STAGE:2022-11-23 01:45:47 24256:24256 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2562530Z STAGE:2022-11-23 01:45:47 24256:24256 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2562892Z STAGE:2022-11-23 01:45:47 24257:24257 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2563315Z STAGE:2022-11-23 01:45:47 24256:24256 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2563697Z STAGE:2022-11-23 01:45:47 24257:24257 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2564060Z STAGE:2022-11-23 01:45:47 24256:24256 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2564423Z STAGE:2022-11-23 01:45:47 24257:24257 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2564791Z STAGE:2022-11-23 01:45:47 24257:24257 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2565156Z STAGE:2022-11-23 01:45:47 24256:24256 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2565538Z STAGE:2022-11-23 01:45:47 24257:24257 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2566131Z STAGE:2022-11-23 01:45:47 24256:24256 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2566278Z ok (4.264s)
2022-11-23T01:48:21.2566298Z 
2022-11-23T01:48:21.2566589Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2566739Z Ran 1 test in 4.264s
2022-11-23T01:48:21.2566759Z 
2022-11-23T01:48:21.2566885Z OK
2022-11-23T01:48:21.2566904Z 
2022-11-23T01:48:21.2567062Z Generating XML reports...
2022-11-23T01:48:21.2567549Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014543.xml
2022-11-23T01:48:21.2568039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2568261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2568679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2568907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2568931Z 
2022-11-23T01:48:21.2569062Z Running tests...
2022-11-23T01:48:21.2569360Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2569711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2569998Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2570252Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24375
2022-11-23T01:48:21.2570504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24376
2022-11-23T01:48:21.2570915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2571128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2571530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2571759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2572161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2572367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2572769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2572992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2573277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2573558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2573995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2574486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2574753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2575015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2575209Z skip: Skipped due to small world size. (4.111s)
2022-11-23T01:48:21.2575228Z 
2022-11-23T01:48:21.2575524Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2575673Z Ran 1 test in 4.111s
2022-11-23T01:48:21.2575693Z 
2022-11-23T01:48:21.2575839Z OK (skipped=1)
2022-11-23T01:48:21.2575858Z 
2022-11-23T01:48:21.2576018Z Generating XML reports...
2022-11-23T01:48:21.2576504Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014550.xml
2022-11-23T01:48:21.2576895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2577112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2577527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2577755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2577775Z 
2022-11-23T01:48:21.2577919Z Running tests...
2022-11-23T01:48:21.2578218Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2578610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2578914Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2579153Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24484
2022-11-23T01:48:21.2579403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24485
2022-11-23T01:48:21.2579821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2580031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2580446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2580670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2581069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2581281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2581692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2581901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2582182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2582458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2582888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2583320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2583582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2583851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2583993Z ok (4.214s)
2022-11-23T01:48:21.2584013Z 
2022-11-23T01:48:21.2584300Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2584451Z Ran 1 test in 4.214s
2022-11-23T01:48:21.2584518Z 
2022-11-23T01:48:21.2584650Z OK
2022-11-23T01:48:21.2584670Z 
2022-11-23T01:48:21.2584829Z Generating XML reports...
2022-11-23T01:48:21.2585314Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014556.xml
2022-11-23T01:48:21.2585720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2585935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2586353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2586579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2586599Z 
2022-11-23T01:48:21.2586729Z Running tests...
2022-11-23T01:48:21.2587029Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2587377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2587659Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2587915Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24593
2022-11-23T01:48:21.2588166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24594
2022-11-23T01:48:21.2588571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2588781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2589232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2589464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2589867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2590079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2590485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2590705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2590986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2591263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2591702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2592123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2592393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2592661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2592800Z ok (4.355s)
2022-11-23T01:48:21.2592820Z 
2022-11-23T01:48:21.2593117Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2593264Z Ran 1 test in 4.355s
2022-11-23T01:48:21.2593284Z 
2022-11-23T01:48:21.2593408Z OK
2022-11-23T01:48:21.2593427Z 
2022-11-23T01:48:21.2593583Z Generating XML reports...
2022-11-23T01:48:21.2594054Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014603.xml
2022-11-23T01:48:21.2594468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2594679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2595097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2595379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2595399Z 
2022-11-23T01:48:21.2595544Z Running tests...
2022-11-23T01:48:21.2595840Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2596186Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2596483Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2596723Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24702
2022-11-23T01:48:21.2596976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24703
2022-11-23T01:48:21.2597384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2597594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2598011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2598232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2598628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2598839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2599249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2599507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2599800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2600078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2600516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2600950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2601214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2601477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2601611Z ok (4.256s)
2022-11-23T01:48:21.2601631Z 
2022-11-23T01:48:21.2601917Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2602072Z Ran 1 test in 4.256s
2022-11-23T01:48:21.2602091Z 
2022-11-23T01:48:21.2602216Z OK
2022-11-23T01:48:21.2602236Z 
2022-11-23T01:48:21.2602393Z Generating XML reports...
2022-11-23T01:48:21.2602883Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014610.xml
2022-11-23T01:48:21.2603290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2603500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2603910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2604132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2604152Z 
2022-11-23T01:48:21.2604283Z Running tests...
2022-11-23T01:48:21.2604582Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2604935Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2605259Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2605512Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24811
2022-11-23T01:48:21.2606023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24812
2022-11-23T01:48:21.2606436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2606647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2607048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2607273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2607676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2607886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2608292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2608516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2608791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2609070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2609508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2609925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2610260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2610645Z STAGE:2022-11-23 01:46:21 24812:24812 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2610904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2611278Z STAGE:2022-11-23 01:46:21 24811:24811 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2611643Z STAGE:2022-11-23 01:46:21 24811:24811 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2612003Z STAGE:2022-11-23 01:46:21 24812:24812 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2612381Z STAGE:2022-11-23 01:46:21 24811:24811 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2612762Z STAGE:2022-11-23 01:46:21 24812:24812 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2612890Z ok (4.307s)
2022-11-23T01:48:21.2612910Z 
2022-11-23T01:48:21.2613212Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2613360Z Ran 1 test in 4.307s
2022-11-23T01:48:21.2613380Z 
2022-11-23T01:48:21.2613505Z OK
2022-11-23T01:48:21.2613524Z 
2022-11-23T01:48:21.2613678Z Generating XML reports...
2022-11-23T01:48:21.2615505Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014617.xml
2022-11-23T01:48:21.2615918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2616131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2616533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2616760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2616780Z 
2022-11-23T01:48:21.2616927Z Running tests...
2022-11-23T01:48:21.2617227Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2617572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2617892Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2618257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24924
2022-11-23T01:48:21.2618511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24925
2022-11-23T01:48:21.2618925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2619122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2619530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2619762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2620167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2620377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2620789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2621013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2621297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2621559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2621997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2622480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2622756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2623019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2623398Z STAGE:2022-11-23 01:46:27 24925:24925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2623757Z STAGE:2022-11-23 01:46:27 24924:24924 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2624167Z STAGE:2022-11-23 01:46:27 24924:24924 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2624758Z STAGE:2022-11-23 01:46:27 24925:24925 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:46:27 24924:24924 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2624779Z 
2022-11-23T01:48:21.2625169Z STAGE:2022-11-23 01:46:27 24925:24925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2625296Z ok (4.116s)
2022-11-23T01:48:21.2625315Z 
2022-11-23T01:48:21.2625614Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2625762Z Ran 1 test in 4.117s
2022-11-23T01:48:21.2625787Z 
2022-11-23T01:48:21.2625913Z OK
2022-11-23T01:48:21.2625933Z 
2022-11-23T01:48:21.2626092Z Generating XML reports...
2022-11-23T01:48:21.2626577Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014624.xml
2022-11-23T01:48:21.2626984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2627197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2627621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2627836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2627856Z 
2022-11-23T01:48:21.2628001Z Running tests...
2022-11-23T01:48:21.2628303Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2628655Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2629065Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2629321Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25037
2022-11-23T01:48:21.2629570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25038
2022-11-23T01:48:21.2629980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2630176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2630595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2630822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2631225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2631442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2631856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2632083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2632364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2632643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2633120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2633564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2633827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2634204Z STAGE:2022-11-23 01:46:34 25038:25038 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2634466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2634822Z STAGE:2022-11-23 01:46:34 25037:25037 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2635189Z STAGE:2022-11-23 01:46:34 25038:25038 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2635577Z STAGE:2022-11-23 01:46:34 25038:25038 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2635953Z STAGE:2022-11-23 01:46:34 25037:25037 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2636321Z STAGE:2022-11-23 01:46:34 25037:25037 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2636460Z ok (4.327s)
2022-11-23T01:48:21.2636484Z 
2022-11-23T01:48:21.2636787Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2636936Z Ran 1 test in 4.328s
2022-11-23T01:48:21.2636956Z 
2022-11-23T01:48:21.2637083Z OK
2022-11-23T01:48:21.2637103Z 
2022-11-23T01:48:21.2637262Z Generating XML reports...
2022-11-23T01:48:21.2637755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014630.xml
2022-11-23T01:48:21.2638162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2638364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2638782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2639009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2639029Z 
2022-11-23T01:48:21.2639174Z Running tests...
2022-11-23T01:48:21.2639531Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2639883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2640154Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T01:48:21.2640175Z 
2022-11-23T01:48:21.2640466Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2640613Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2640633Z 
2022-11-23T01:48:21.2640807Z OK (skipped=1)
2022-11-23T01:48:21.2640828Z 
2022-11-23T01:48:21.2640981Z Generating XML reports...
2022-11-23T01:48:21.2641473Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014637.xml
2022-11-23T01:48:21.2641880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2642094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2642510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2642734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2642754Z 
2022-11-23T01:48:21.2642900Z Running tests...
2022-11-23T01:48:21.2643182Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2643530Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2643896Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T01:48:21.2643919Z 
2022-11-23T01:48:21.2644226Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2644371Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2644391Z 
2022-11-23T01:48:21.2644535Z OK (skipped=1)
2022-11-23T01:48:21.2644559Z 
2022-11-23T01:48:21.2644716Z Generating XML reports...
2022-11-23T01:48:21.2645200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014639.xml
2022-11-23T01:48:21.2645605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2646053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2646479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2646708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2646728Z 
2022-11-23T01:48:21.2646871Z Running tests...
2022-11-23T01:48:21.2647167Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2647515Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2647810Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T01:48:21.2647830Z 
2022-11-23T01:48:21.2648123Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2648269Z Ran 1 test in 0.002s
2022-11-23T01:48:21.2648289Z 
2022-11-23T01:48:21.2648418Z OK (skipped=1)
2022-11-23T01:48:21.2648437Z 
2022-11-23T01:48:21.2648595Z Generating XML reports...
2022-11-23T01:48:21.2649076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014642.xml
2022-11-23T01:48:21.2649484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2649694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2650109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2650419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2650439Z 
2022-11-23T01:48:21.2650583Z Running tests...
2022-11-23T01:48:21.2650884Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2651217Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2651525Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2651779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25249
2022-11-23T01:48:21.2652039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25250
2022-11-23T01:48:21.2652443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2652648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2653064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2653293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2653685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2653893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2654302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2654523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2654858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2655141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2655583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2656015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2656275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2656523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2656895Z STAGE:2022-11-23 01:46:48 25249:25249 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2657265Z STAGE:2022-11-23 01:46:48 25250:25250 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2657636Z STAGE:2022-11-23 01:46:48 25250:25250 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2658017Z STAGE:2022-11-23 01:46:48 25250:25250 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2658382Z STAGE:2022-11-23 01:46:48 25249:25249 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2658767Z STAGE:2022-11-23 01:46:48 25249:25249 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2658905Z ok (4.290s)
2022-11-23T01:48:21.2658926Z 
2022-11-23T01:48:21.2659224Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2659359Z Ran 1 test in 4.290s
2022-11-23T01:48:21.2659378Z 
2022-11-23T01:48:21.2659505Z OK
2022-11-23T01:48:21.2659524Z 
2022-11-23T01:48:21.2659679Z Generating XML reports...
2022-11-23T01:48:21.2660170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014644.xml
2022-11-23T01:48:21.2660577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2660788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2661292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2661519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2661539Z 
2022-11-23T01:48:21.2661668Z Running tests...
2022-11-23T01:48:21.2661966Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2662315Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2662606Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2662864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25362
2022-11-23T01:48:21.2663116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25363
2022-11-23T01:48:21.2663520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2663733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2664149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2664361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2664761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2664965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2665419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2665651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2665929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2666209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2666651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2667067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2667332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2667596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2667733Z ok (4.335s)
2022-11-23T01:48:21.2667754Z 
2022-11-23T01:48:21.2668055Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2668204Z Ran 1 test in 4.335s
2022-11-23T01:48:21.2668224Z 
2022-11-23T01:48:21.2668350Z OK
2022-11-23T01:48:21.2668369Z 
2022-11-23T01:48:21.2668529Z Generating XML reports...
2022-11-23T01:48:21.2669015Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014651.xml
2022-11-23T01:48:21.2669414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2669624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2670040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2670261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2670281Z 
2022-11-23T01:48:21.2670425Z Running tests...
2022-11-23T01:48:21.2670725Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2671071Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2671394Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2671685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25471
2022-11-23T01:48:21.2671938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25472
2022-11-23T01:48:21.2672342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2672552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2672969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2673196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2673594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2673801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2674211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2674423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2674706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2674984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2675419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2675896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2676166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2676540Z STAGE:2022-11-23 01:47:02 25472:25472 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2676807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2677176Z STAGE:2022-11-23 01:47:02 25471:25471 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2677535Z STAGE:2022-11-23 01:47:02 25471:25471 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2677918Z STAGE:2022-11-23 01:47:02 25471:25471 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2678289Z STAGE:2022-11-23 01:47:02 25472:25472 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2678672Z STAGE:2022-11-23 01:47:02 25472:25472 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2678811Z ok (4.353s)
2022-11-23T01:48:21.2678832Z 
2022-11-23T01:48:21.2679125Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2679274Z Ran 1 test in 4.353s
2022-11-23T01:48:21.2679294Z 
2022-11-23T01:48:21.2679425Z OK
2022-11-23T01:48:21.2679445Z 
2022-11-23T01:48:21.2679589Z Generating XML reports...
2022-11-23T01:48:21.2680071Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014658.xml
2022-11-23T01:48:21.2680474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2680728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2681187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2681459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2681479Z 
2022-11-23T01:48:21.2681661Z Running tests...
2022-11-23T01:48:21.2682127Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2682472Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2682882Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2683173Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25584
2022-11-23T01:48:21.2683463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25585
2022-11-23T01:48:21.2691607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2691866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2692325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2692560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2692973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2693190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2693608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2693839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2694121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2694404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2694973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2695408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2695681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2696069Z STAGE:2022-11-23 01:47:09 25585:25585 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2696335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2696701Z STAGE:2022-11-23 01:47:09 25584:25584 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:48:21.2697072Z STAGE:2022-11-23 01:47:09 25584:25584 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2697459Z STAGE:2022-11-23 01:47:09 25584:25584 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2697833Z STAGE:2022-11-23 01:47:09 25585:25585 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:48:21.2698214Z STAGE:2022-11-23 01:47:09 25585:25585 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:48:21.2698337Z ok (4.239s)
2022-11-23T01:48:21.2698360Z 
2022-11-23T01:48:21.2698662Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2698820Z Ran 1 test in 4.240s
2022-11-23T01:48:21.2698840Z 
2022-11-23T01:48:21.2698969Z OK
2022-11-23T01:48:21.2698988Z 
2022-11-23T01:48:21.2699148Z Generating XML reports...
2022-11-23T01:48:21.2699642Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014705.xml
2022-11-23T01:48:21.2700049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2700266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2700688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2700905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2700924Z 
2022-11-23T01:48:21.2701071Z Running tests...
2022-11-23T01:48:21.2701373Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2701803Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2702107Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2702362Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25697
2022-11-23T01:48:21.2702614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25698
2022-11-23T01:48:21.2703025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2703227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2703651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2703877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2704284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2704498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2704910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2705134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2705416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2705730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2706181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2706614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2706885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2707146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2707285Z ok (4.232s)
2022-11-23T01:48:21.2707306Z 
2022-11-23T01:48:21.2707607Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2707758Z Ran 1 test in 4.232s
2022-11-23T01:48:21.2707778Z 
2022-11-23T01:48:21.2707908Z OK
2022-11-23T01:48:21.2707927Z 
2022-11-23T01:48:21.2708071Z Generating XML reports...
2022-11-23T01:48:21.2708563Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014712.xml
2022-11-23T01:48:21.2708971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2709185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2709605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2709829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2709850Z 
2022-11-23T01:48:21.2709991Z Running tests...
2022-11-23T01:48:21.2710293Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2710647Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2710941Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2711202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25896
2022-11-23T01:48:21.2711459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25897
2022-11-23T01:48:21.2711869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2712133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2712552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2712780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2713180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2713371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2713787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2714014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2714299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2714580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2715027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2715461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2715727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2715994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2716118Z ok (5.049s)
2022-11-23T01:48:21.2716239Z 
2022-11-23T01:48:21.2716548Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2716698Z Ran 1 test in 5.049s
2022-11-23T01:48:21.2716718Z 
2022-11-23T01:48:21.2716847Z OK
2022-11-23T01:48:21.2716867Z 
2022-11-23T01:48:21.2717024Z Generating XML reports...
2022-11-23T01:48:21.2717518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014719.xml
2022-11-23T01:48:21.2717927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2718139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2718539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2718769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2718789Z 
2022-11-23T01:48:21.2718935Z Running tests...
2022-11-23T01:48:21.2719232Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2719580Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2719880Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2720138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26097
2022-11-23T01:48:21.2720393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26098
2022-11-23T01:48:21.2720801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2720997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2721412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2721643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2722043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2722256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2722667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2722951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2723235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2723501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2723942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2724381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2724651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2724915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2725056Z ok (5.560s)
2022-11-23T01:48:21.2725076Z 
2022-11-23T01:48:21.2725380Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2725530Z Ran 1 test in 5.560s
2022-11-23T01:48:21.2725550Z 
2022-11-23T01:48:21.2725950Z OK
2022-11-23T01:48:21.2725972Z 
2022-11-23T01:48:21.2726127Z Generating XML reports...
2022-11-23T01:48:21.2726626Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014726.xml
2022-11-23T01:48:21.2727038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2727333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2727761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2727988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2728009Z 
2022-11-23T01:48:21.2728160Z Running tests...
2022-11-23T01:48:21.2728462Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2728812Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2729095Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2729352Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26212
2022-11-23T01:48:21.2729602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26213
2022-11-23T01:48:21.2730015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2730227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2730646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2730875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2731276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2731470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2731880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2732104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2732393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2732674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2733116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2733547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2733890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2734153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2734429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8yh0o8s
2022-11-23T01:48:21.2734735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8yh0o8s/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2735031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpny4eksgj
2022-11-23T01:48:21.2735336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpny4eksgj/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2735474Z ok (4.261s)
2022-11-23T01:48:21.2735494Z 
2022-11-23T01:48:21.2735798Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2735953Z Ran 1 test in 4.262s
2022-11-23T01:48:21.2735973Z 
2022-11-23T01:48:21.2736100Z OK
2022-11-23T01:48:21.2736118Z 
2022-11-23T01:48:21.2736263Z Generating XML reports...
2022-11-23T01:48:21.2736757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014734.xml
2022-11-23T01:48:21.2737164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2737378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2737844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2738080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2738099Z 
2022-11-23T01:48:21.2738244Z Running tests...
2022-11-23T01:48:21.2738549Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2738902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2739176Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2739431Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26325
2022-11-23T01:48:21.2739683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26326
2022-11-23T01:48:21.2740092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2740308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2740764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2740994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2741395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2741593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2742006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2742231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2742513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2742792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2743236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2743674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2743940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2744268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2744545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuz7k2qz
2022-11-23T01:48:21.2744854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuz7k2qz/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2745144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobm3xutk
2022-11-23T01:48:21.2745454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobm3xutk/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2745596Z ok (5.055s)
2022-11-23T01:48:21.2745616Z 
2022-11-23T01:48:21.2745920Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2746065Z Ran 1 test in 5.055s
2022-11-23T01:48:21.2746084Z 
2022-11-23T01:48:21.2746212Z OK
2022-11-23T01:48:21.2746231Z 
2022-11-23T01:48:21.2746393Z Generating XML reports...
2022-11-23T01:48:21.2746871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014741.xml
2022-11-23T01:48:21.2747281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2747495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2747912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2748137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2748207Z 
2022-11-23T01:48:21.2748361Z Running tests...
2022-11-23T01:48:21.2748664Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2749012Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2749331Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2749592Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26436
2022-11-23T01:48:21.2749844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26437
2022-11-23T01:48:21.2750258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2750469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2750893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2751119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2751522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2751732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2752131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2752355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2752637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2752916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2753358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2753795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2754061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2754324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2754675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsy7vcn2i
2022-11-23T01:48:21.2754968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsy7vcn2i/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2755260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjb755nxa
2022-11-23T01:48:21.2755566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjb755nxa/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2756388Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:21.2757212Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:48:21.2757400Z ok (5.576s)
2022-11-23T01:48:21.2757424Z 
2022-11-23T01:48:21.2757732Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2757882Z Ran 1 test in 5.576s
2022-11-23T01:48:21.2757902Z 
2022-11-23T01:48:21.2758029Z OK
2022-11-23T01:48:21.2758048Z 
2022-11-23T01:48:21.2758207Z Generating XML reports...
2022-11-23T01:48:21.2758705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014749.xml
2022-11-23T01:48:21.2759121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2759319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2759738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2759964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2759984Z 
2022-11-23T01:48:21.2760129Z Running tests...
2022-11-23T01:48:21.2760430Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2760783Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2761101Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2761362Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26551
2022-11-23T01:48:21.2761601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26552
2022-11-23T01:48:21.2762014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2762226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2762635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2762868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2763268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2763476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2763945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2764175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2764441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2764723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2765165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2765599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2766097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2766368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2766648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2766922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2767366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2767778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2768138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:21.2768418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:21.2768847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.2769277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.2769576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52giukpz
2022-11-23T01:48:21.2769884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52giukpz/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2770175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkkd3_ft
2022-11-23T01:48:21.2770480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkkd3_ft/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2770604Z ok (10.229s)
2022-11-23T01:48:21.2770630Z 
2022-11-23T01:48:21.2770936Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2771091Z Ran 1 test in 10.229s
2022-11-23T01:48:21.2771111Z 
2022-11-23T01:48:21.2771239Z OK
2022-11-23T01:48:21.2771259Z 
2022-11-23T01:48:21.2771418Z Generating XML reports...
2022-11-23T01:48:21.2771915Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014757.xml
2022-11-23T01:48:21.2772324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2772536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2772938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2773172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2773193Z 
2022-11-23T01:48:21.2773344Z Running tests...
2022-11-23T01:48:21.2773644Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2773993Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:48:21.2774317Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:48:21.2774642Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26674
2022-11-23T01:48:21.2774901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26675
2022-11-23T01:48:21.2775312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2775506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2775923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2776155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2776560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:48:21.2776770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:48:21.2777185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:48:21.2777415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:48:21.2777695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:48:21.2777959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:48:21.2778398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2778878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:48:21.2779155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:48:21.2779417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:48:21.2779696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:48:21.2779973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:48:21.2780411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2780838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:48:21.2781113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:48:21.2781374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:48:21.2781803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.2782238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:48:21.2782538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpenk7_71s
2022-11-23T01:48:21.2782845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpenk7_71s/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2783136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0k0a8k60
2022-11-23T01:48:21.2783439Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0k0a8k60/_remote_module_non_scriptable.py
2022-11-23T01:48:21.2783577Z ok (10.176s)
2022-11-23T01:48:21.2783597Z 
2022-11-23T01:48:21.2783904Z ----------------------------------------------------------------------
2022-11-23T01:48:21.2784040Z Ran 1 test in 10.176s
2022-11-23T01:48:21.2784059Z 
2022-11-23T01:48:21.2784187Z OK
2022-11-23T01:48:21.2784206Z 
2022-11-23T01:48:21.2784365Z Generating XML reports...
2022-11-23T01:48:21.2784854Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014810.xml
2022-11-23T01:48:21.2784920Z 
2022-11-23T01:48:21.2785670Z ##[endgroup]
2022-11-23T01:48:21.2786190Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_gcmccc03)
2022-11-23T01:48:21.2786211Z 
2022-11-23T01:48:21.2786460Z Running distributed tests for the gloo backend with file init_method in shard 1 of 3
2022-11-23T01:48:21.2787016Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:48:20.937000]
2022-11-23T02:16:44.1399398Z 
2022-11-23T02:16:44.1401875Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T02:16:44.1402965Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_vzl_fg2e)
2022-11-23T02:16:44.1403561Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:16:44.1460073Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-11-23T02:16:44.1522195Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1523373Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1526980Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1527920Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1528965Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1529630Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1530196Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1530835Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1531840Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1532936Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1534015Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1535043Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1536056Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1537063Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1538184Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1539141Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1540025Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1540915Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1541763Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1542675Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1543615Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1544539Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1545370Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1546128Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1546937Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1547768Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1548614Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1549440Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1550272Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1551039Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1551809Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1552594Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1553393Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1554186Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1555042Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1555859Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1556680Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1557490Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1558323Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1559281Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1560104Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1560956Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1561811Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1562673Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1563443Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1564235Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1565106Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1566421Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1567222Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1568084Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1568945Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1569714Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1570510Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1571334Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1572182Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1573089Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1573940Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1574725Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1575475Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1576273Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1577038Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1577866Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1578545Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1579264Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1580013Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1580779Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1581561Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1582338Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1583063Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1583795Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1584578Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1585291Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1586012Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1586813Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1587543Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1588240Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1588988Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1589710Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1590458Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1591177Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1591903Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1592619Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1593494Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1594324Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1595366Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1596070Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1596578Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1597095Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1597612Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1598104Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1598598Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1599105Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1599609Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1600102Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1600627Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1601146Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1632020Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1632700Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1633205Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1633664Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1634106Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1634524Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1634946Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1635389Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1635837Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1636290Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1636737Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1637187Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1637667Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1638133Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1638595Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1639049Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1639544Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1640027Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1640490Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1640971Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1641450Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1641946Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1642418Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1642895Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1643345Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1643767Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1644216Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1644756Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1645193Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1645652Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1646776Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1647358Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1647859Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1648340Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1648821Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1649303Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1649821Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1650359Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1650864Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1651335Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1651842Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1652315Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1652737Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1653276Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1653766Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1654227Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1654725Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1655232Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1655711Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1656174Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1656698Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1657265Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1657883Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1658563Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1659237Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1659919Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1660603Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1661260Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1661937Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1662612Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1663229Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1663856Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1664364Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1664813Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1665258Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1665729Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1666183Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1666654Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1667161Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1667687Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1668214Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1668682Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1669135Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1669611Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1670095Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1670591Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1671071Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1671548Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1672077Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1672586Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1673077Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1673538Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1674020Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1674504Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1674979Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1675456Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1675985Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1676496Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1676954Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1677410Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1677881Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1678355Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1678847Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1679293Z test_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1679726Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1680146Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1680596Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1681043Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1681468Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1681929Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1682385Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1682802Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1683228Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1683679Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1684208Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1684653Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1685093Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1685510Z test_isend (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1686450Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1686931Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1687414Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1687928Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1688448Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1688933Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1689425Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1689921Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1690417Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1690907Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1691379Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1691860Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1692333Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1692883Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1693348Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1693815Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1694352Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1694904Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1695444Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1695963Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1696476Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1697005Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1697505Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1697997Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1698481Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1698988Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1699507Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1700047Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1700635Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1701191Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1701679Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1702128Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1702599Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1703076Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1703577Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1704030Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1704488Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1705009Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1705448Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1705876Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1706319Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1706755Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1707219Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1707691Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1708123Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1708563Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1709015Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1709453Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1709894Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1710330Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1710776Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1711201Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1711652Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1712114Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1712549Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1713000Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1713488Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1713929Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1714422Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1714935Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1715440Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1715886Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1716400Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1716893Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1717360Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1717824Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1718309Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1718798Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1719281Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1719749Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1720223Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1720688Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1721137Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1721625Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1722126Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1722638Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.1723436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1723966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1724606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1725137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1725463Z 
2022-11-23T02:16:44.1725599Z Running tests...
2022-11-23T02:16:44.1726633Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1727231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1727875Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1728488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26830
2022-11-23T02:16:44.1728987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26831
2022-11-23T02:16:44.1729652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1730170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1730798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1731346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1731995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1732503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1733122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1733657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1734264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.1734849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.1735565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1736327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1736924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.1737451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.1738050Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:16:44.1738960Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:16:44.1739697Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:16:44.1740570Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:16:44.1741303Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:16:44.1741940Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:16:44.1742834Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:16:44.1743814Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:16:44.1744538Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:16:44.1745168Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:16:44.1746155Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:16:44.1747134Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:16:44.1747842Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:16:44.1748490Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:16:44.1749380Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:16:44.1750353Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:16:44.1750900Z ok (5.667s)
2022-11-23T02:16:44.1751084Z 
2022-11-23T02:16:44.1751389Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1751787Z Ran 1 test in 5.667s
2022-11-23T02:16:44.1751984Z 
2022-11-23T02:16:44.1752095Z OK
2022-11-23T02:16:44.1752265Z 
2022-11-23T02:16:44.1752422Z Generating XML reports...
2022-11-23T02:16:44.1753102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014825.xml
2022-11-23T02:16:44.1753943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1754459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1755108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1755653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1755921Z 
2022-11-23T02:16:44.1756066Z Running tests...
2022-11-23T02:16:44.1756524Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1757125Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1757723Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-11-23T02:16:44.1758067Z 
2022-11-23T02:16:44.1758351Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1758747Z Ran 1 test in 0.004s
2022-11-23T02:16:44.1758941Z 
2022-11-23T02:16:44.1759084Z OK (skipped=1)
2022-11-23T02:16:44.1759272Z 
2022-11-23T02:16:44.1759429Z Generating XML reports...
2022-11-23T02:16:44.1760085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014833.xml
2022-11-23T02:16:44.1760878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1761397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1762025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1762568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1762830Z 
2022-11-23T02:16:44.1762972Z Running tests...
2022-11-23T02:16:44.1763447Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1764034Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1764610Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1765170Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26975
2022-11-23T02:16:44.1766203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26976
2022-11-23T02:16:44.1766934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1767452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1768087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1768606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1769257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1769774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1770395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1770936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1771455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.1772029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.1772736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1773494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1774174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.1774730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.1775128Z ok (4.265s)
2022-11-23T02:16:44.1775313Z 
2022-11-23T02:16:44.1775619Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1776022Z Ran 1 test in 4.265s
2022-11-23T02:16:44.1776216Z 
2022-11-23T02:16:44.1776325Z OK
2022-11-23T02:16:44.1776493Z 
2022-11-23T02:16:44.1776648Z Generating XML reports...
2022-11-23T02:16:44.1777325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014835.xml
2022-11-23T02:16:44.1778114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1778619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1779266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1779805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1780069Z 
2022-11-23T02:16:44.1780196Z Running tests...
2022-11-23T02:16:44.1780669Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1781274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1781870Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1782978Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.626s)
2022-11-23T02:16:44.1783561Z 
2022-11-23T02:16:44.1783847Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1784244Z Ran 1 test in 1.626s
2022-11-23T02:16:44.1784438Z 
2022-11-23T02:16:44.1784581Z OK (skipped=1)
2022-11-23T02:16:44.1784770Z 
2022-11-23T02:16:44.1784910Z Generating XML reports...
2022-11-23T02:16:44.1785588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014842.xml
2022-11-23T02:16:44.1786456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1786975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1787607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1788147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1788411Z 
2022-11-23T02:16:44.1788554Z Running tests...
2022-11-23T02:16:44.1789031Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1789618Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1790229Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1790824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27118
2022-11-23T02:16:44.1791331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27119
2022-11-23T02:16:44.1792004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1792525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1793169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1793752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1794411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1794922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1795548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1796089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1796613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.1797181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.1797894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1798651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1799240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.1799781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.1800337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4sj69xf
2022-11-23T02:16:44.1800927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppm5kg9ir
2022-11-23T02:16:44.1801525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4sj69xf/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1802128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppm5kg9ir/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1802712Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1803270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1803680Z ok (4.349s)
2022-11-23T02:16:44.1803846Z 
2022-11-23T02:16:44.1804157Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1804554Z Ran 1 test in 4.350s
2022-11-23T02:16:44.1804747Z 
2022-11-23T02:16:44.1804874Z OK
2022-11-23T02:16:44.1805043Z 
2022-11-23T02:16:44.1805245Z Generating XML reports...
2022-11-23T02:16:44.1806514Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014846.xml
2022-11-23T02:16:44.1807325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1807843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1808470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1809008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1809275Z 
2022-11-23T02:16:44.1809418Z Running tests...
2022-11-23T02:16:44.1809869Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1810467Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1811090Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1811701Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27231
2022-11-23T02:16:44.1812205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27232
2022-11-23T02:16:44.1812883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1813400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1814127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1814663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1815319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1815829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1816495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1817030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1817553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.1818126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.1818840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1819599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1820193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.1820736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.1821295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3kepqvka
2022-11-23T02:16:44.1821889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpka_ozyyc
2022-11-23T02:16:44.1822492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3kepqvka/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1823092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpka_ozyyc/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1823678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1824236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1824659Z ok (4.245s)
2022-11-23T02:16:44.1824827Z 
2022-11-23T02:16:44.1825133Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1825528Z Ran 1 test in 4.246s
2022-11-23T02:16:44.1825798Z 
2022-11-23T02:16:44.1825924Z OK
2022-11-23T02:16:44.1826092Z 
2022-11-23T02:16:44.1826232Z Generating XML reports...
2022-11-23T02:16:44.1826909Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014853.xml
2022-11-23T02:16:44.1827689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1828209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1828845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1829383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1829647Z 
2022-11-23T02:16:44.1829789Z Running tests...
2022-11-23T02:16:44.1830243Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1830840Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1831465Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1832068Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27344
2022-11-23T02:16:44.1832571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27345
2022-11-23T02:16:44.1833244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1833761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1834435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1834987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1835636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1836151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1836767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1837302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1837830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.1838403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.1839118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1839877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1840469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.1840998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.1841566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9gsx7ck
2022-11-23T02:16:44.1842175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9gsx7ck/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1842777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbkt6dw6
2022-11-23T02:16:44.1843370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbkt6dw6/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1843955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1844509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1845061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1845658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1846801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1847346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1847874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1848419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1848965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1849503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1850038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1850578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1850995Z ok (6.072s)
2022-11-23T02:16:44.1851180Z 
2022-11-23T02:16:44.1851481Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1851879Z Ran 1 test in 6.073s
2022-11-23T02:16:44.1852073Z 
2022-11-23T02:16:44.1852201Z OK
2022-11-23T02:16:44.1852369Z 
2022-11-23T02:16:44.1852527Z Generating XML reports...
2022-11-23T02:16:44.1853190Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014900.xml
2022-11-23T02:16:44.1854062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1854597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1855226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1855765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1856037Z 
2022-11-23T02:16:44.1856183Z Running tests...
2022-11-23T02:16:44.1856658Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1857240Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1857873Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1858482Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27459
2022-11-23T02:16:44.1858993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27460
2022-11-23T02:16:44.1859670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1860192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1860836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1861360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1862007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1862521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1863139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1863672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1864194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.1864765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.1865478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1866317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1866912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.1867452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.1868006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprxjt0qm
2022-11-23T02:16:44.1868619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprxjt0qm/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1869224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_k7jtyh
2022-11-23T02:16:44.1869812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_k7jtyh/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1870397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1870957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1871502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1872036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1872451Z ok (5.140s)
2022-11-23T02:16:44.1872633Z 
2022-11-23T02:16:44.1872941Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1873324Z Ran 1 test in 5.140s
2022-11-23T02:16:44.1873520Z 
2022-11-23T02:16:44.1873649Z OK
2022-11-23T02:16:44.1873818Z 
2022-11-23T02:16:44.1874027Z Generating XML reports...
2022-11-23T02:16:44.1874710Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014909.xml
2022-11-23T02:16:44.1875480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1876007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1876654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1877194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1877442Z 
2022-11-23T02:16:44.1877589Z Running tests...
2022-11-23T02:16:44.1878064Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1878662Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1879288Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1879908Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27574
2022-11-23T02:16:44.1880430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27575
2022-11-23T02:16:44.1881114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1881615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1882256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1882790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1883417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1883935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1884573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1885105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1885970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.1886558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.1887295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1888056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1888631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.1889178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.1889748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hrf3edm
2022-11-23T02:16:44.1890343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hrf3edm/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1890950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqofg5sy9
2022-11-23T02:16:44.1891553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqofg5sy9/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1892137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1892678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1893230Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1893852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1894395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1894945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1895490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1896035Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1896435Z ok (5.244s)
2022-11-23T02:16:44.1896622Z 
2022-11-23T02:16:44.1896931Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1897330Z Ran 1 test in 5.244s
2022-11-23T02:16:44.1897526Z 
2022-11-23T02:16:44.1897637Z OK
2022-11-23T02:16:44.1897806Z 
2022-11-23T02:16:44.1897963Z Generating XML reports...
2022-11-23T02:16:44.1898649Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014916.xml
2022-11-23T02:16:44.1899443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1899947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1900587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1901131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1901397Z 
2022-11-23T02:16:44.1901540Z Running tests...
2022-11-23T02:16:44.1901996Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1902596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1903258Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1903882Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27689
2022-11-23T02:16:44.1904402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27690
2022-11-23T02:16:44.1905077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1905672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1906303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1906838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1907481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1907976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1908620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1909155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1909671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.1910223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.1910949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1911710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1912303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.1912828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.1913439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpowcmqnes
2022-11-23T02:16:44.1914061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpowcmqnes/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1914649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcjlb9xh
2022-11-23T02:16:44.1915256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcjlb9xh/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1915847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1916446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1916850Z ok (5.249s)
2022-11-23T02:16:44.1917031Z 
2022-11-23T02:16:44.1917338Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1917738Z Ran 1 test in 5.249s
2022-11-23T02:16:44.1917931Z 
2022-11-23T02:16:44.1918041Z OK
2022-11-23T02:16:44.1918210Z 
2022-11-23T02:16:44.1918367Z Generating XML reports...
2022-11-23T02:16:44.1919049Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014924.xml
2022-11-23T02:16:44.1919839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1920346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1920995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1921532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1921796Z 
2022-11-23T02:16:44.1921923Z Running tests...
2022-11-23T02:16:44.1922394Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1922993Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1923654Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1924272Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27804
2022-11-23T02:16:44.1924792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27805
2022-11-23T02:16:44.1925467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1926336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1926977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1927514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1928163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1928662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1929310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1929844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1930366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.1930925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.1931652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1932406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1933000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.1933601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.1934177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbyr2w1wb
2022-11-23T02:16:44.1934788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbyr2w1wb/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1935373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotso_s08
2022-11-23T02:16:44.1935984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotso_s08/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1936570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1937122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1937649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1938201Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1938621Z ok (5.747s)
2022-11-23T02:16:44.1938809Z 
2022-11-23T02:16:44.1939102Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1939500Z Ran 1 test in 5.747s
2022-11-23T02:16:44.1939695Z 
2022-11-23T02:16:44.1939821Z OK
2022-11-23T02:16:44.1939983Z 
2022-11-23T02:16:44.1940142Z Generating XML reports...
2022-11-23T02:16:44.1940810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014932.xml
2022-11-23T02:16:44.1941588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1942107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1942736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1943278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1943543Z 
2022-11-23T02:16:44.1943691Z Running tests...
2022-11-23T02:16:44.1944165Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1944748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1945384Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1946072Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27919
2022-11-23T02:16:44.1946575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27920
2022-11-23T02:16:44.1947256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1947773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1948417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1948941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1949587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1950099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1950747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1951270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1951797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.1952365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.1953073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1953879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1954480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.1955026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.1955586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswt_k4rf
2022-11-23T02:16:44.1956194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswt_k4rf/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1956800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplm99dug7
2022-11-23T02:16:44.1957405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplm99dug7/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1957972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1958530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1959080Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1959610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1960024Z ok (5.648s)
2022-11-23T02:16:44.1960209Z 
2022-11-23T02:16:44.1960519Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1960908Z Ran 1 test in 5.648s
2022-11-23T02:16:44.1961086Z 
2022-11-23T02:16:44.1961214Z OK
2022-11-23T02:16:44.1961382Z 
2022-11-23T02:16:44.1961539Z Generating XML reports...
2022-11-23T02:16:44.1962214Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014940.xml
2022-11-23T02:16:44.1962984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1963508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1964150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1964689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1965000Z 
2022-11-23T02:16:44.1965146Z Running tests...
2022-11-23T02:16:44.1965619Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1966486Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1967130Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1967765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28034
2022-11-23T02:16:44.1968281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28035
2022-11-23T02:16:44.1968962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1969469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1970114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1970657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1971291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1971804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1972445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1972976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1973558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.1974145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.1974876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1975638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.1976209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.1976748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.1977320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4oabvqh2
2022-11-23T02:16:44.1977915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4oabvqh2/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1978524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9r3p0fc
2022-11-23T02:16:44.1979131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9r3p0fc/_remote_module_non_scriptable.py
2022-11-23T02:16:44.1979718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1980265Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1980816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1981369Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.1981788Z ok (5.152s)
2022-11-23T02:16:44.1981956Z 
2022-11-23T02:16:44.1982265Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1982665Z Ran 1 test in 5.152s
2022-11-23T02:16:44.1982860Z 
2022-11-23T02:16:44.1982986Z OK
2022-11-23T02:16:44.1983137Z 
2022-11-23T02:16:44.1983297Z Generating XML reports...
2022-11-23T02:16:44.1983973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014949.xml
2022-11-23T02:16:44.1984757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1985355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1985986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1986526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1986790Z 
2022-11-23T02:16:44.1986932Z Running tests...
2022-11-23T02:16:44.1987387Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1987986Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1988613Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1989757Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.630s)
2022-11-23T02:16:44.1990316Z 
2022-11-23T02:16:44.1990600Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1990996Z Ran 1 test in 1.630s
2022-11-23T02:16:44.1991191Z 
2022-11-23T02:16:44.1991336Z OK (skipped=1)
2022-11-23T02:16:44.1991524Z 
2022-11-23T02:16:44.1991680Z Generating XML reports...
2022-11-23T02:16:44.1992334Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014956.xml
2022-11-23T02:16:44.1993167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1993698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1994327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.1994866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.1995136Z 
2022-11-23T02:16:44.1995279Z Running tests...
2022-11-23T02:16:44.1995746Z ----------------------------------------------------------------------
2022-11-23T02:16:44.1996328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.1996944Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.1997539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28183
2022-11-23T02:16:44.1998047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28184
2022-11-23T02:16:44.1998728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.1999249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.1999890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2000419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2001068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2001582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2002226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2002745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2003322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2003894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2004609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2005429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2006284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2006831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2007220Z ok (4.359s)
2022-11-23T02:16:44.2007403Z 
2022-11-23T02:16:44.2007712Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2008108Z Ran 1 test in 4.360s
2022-11-23T02:16:44.2008309Z 
2022-11-23T02:16:44.2008419Z OK
2022-11-23T02:16:44.2008585Z 
2022-11-23T02:16:44.2008743Z Generating XML reports...
2022-11-23T02:16:44.2009419Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015000.xml
2022-11-23T02:16:44.2010203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2010714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2011360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2011898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2012162Z 
2022-11-23T02:16:44.2012306Z Running tests...
2022-11-23T02:16:44.2012758Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2013430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2014074Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2015194Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.607s)
2022-11-23T02:16:44.2015752Z 
2022-11-23T02:16:44.2016046Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2016480Z Ran 1 test in 1.608s
2022-11-23T02:16:44.2016677Z 
2022-11-23T02:16:44.2016820Z OK (skipped=1)
2022-11-23T02:16:44.2017004Z 
2022-11-23T02:16:44.2017143Z Generating XML reports...
2022-11-23T02:16:44.2017821Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015007.xml
2022-11-23T02:16:44.2018598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2019117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2019741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2020284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2020548Z 
2022-11-23T02:16:44.2020692Z Running tests...
2022-11-23T02:16:44.2021142Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2021737Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2022334Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2022911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28326
2022-11-23T02:16:44.2023418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28327
2022-11-23T02:16:44.2024089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2024609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2025324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2025863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2026508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2027017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2027636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2028172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2028697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2029265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2029985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2030745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2031341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2031882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2032276Z ok (5.002s)
2022-11-23T02:16:44.2032459Z 
2022-11-23T02:16:44.2032808Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2033202Z Ran 1 test in 5.002s
2022-11-23T02:16:44.2033382Z 
2022-11-23T02:16:44.2033509Z OK
2022-11-23T02:16:44.2033679Z 
2022-11-23T02:16:44.2033837Z Generating XML reports...
2022-11-23T02:16:44.2034513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015011.xml
2022-11-23T02:16:44.2035293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2035810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2036451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2036989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2037253Z 
2022-11-23T02:16:44.2037382Z Running tests...
2022-11-23T02:16:44.2037855Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2038455Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2039016Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-11-23T02:16:44.2039339Z 
2022-11-23T02:16:44.2039642Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2040038Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2040232Z 
2022-11-23T02:16:44.2040374Z OK (skipped=1)
2022-11-23T02:16:44.2040563Z 
2022-11-23T02:16:44.2040700Z Generating XML reports...
2022-11-23T02:16:44.2041369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015019.xml
2022-11-23T02:16:44.2042151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2042674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2043303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2043842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2044108Z 
2022-11-23T02:16:44.2044250Z Running tests...
2022-11-23T02:16:44.2044767Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2045362Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2046165Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.2046717Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2047248Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28470
2022-11-23T02:16:44.2047767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28471
2022-11-23T02:16:44.2048452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2048958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2049604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2050152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2050797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2051292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2051933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2052463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2053045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2053628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2054356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2055121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2055696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2056238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2056799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxmqkwxf
2022-11-23T02:16:44.2057393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjf592igf
2022-11-23T02:16:44.2057985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxmqkwxf/_remote_module_non_scriptable.py
2022-11-23T02:16:44.2058604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjf592igf/_remote_module_non_scriptable.py
2022-11-23T02:16:44.2059054Z ok (4.249s)
2022-11-23T02:16:44.2059243Z 
2022-11-23T02:16:44.2059526Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2059931Z Ran 1 test in 4.249s
2022-11-23T02:16:44.2060128Z 
2022-11-23T02:16:44.2060255Z OK
2022-11-23T02:16:44.2060424Z 
2022-11-23T02:16:44.2060581Z Generating XML reports...
2022-11-23T02:16:44.2061246Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015021.xml
2022-11-23T02:16:44.2062031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2062550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2063181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2063725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2063988Z 
2022-11-23T02:16:44.2064132Z Running tests...
2022-11-23T02:16:44.2064604Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2065265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2065815Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.2066384Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2066925Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28583
2022-11-23T02:16:44.2067446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28584
2022-11-23T02:16:44.2068121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2068641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2069265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2069811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2070456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2070971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2071590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2072123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2072693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2073256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2073989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2074746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2075344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2075896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgafql11
2022-11-23T02:16:44.2076500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgafql11/_remote_module_non_scriptable.py
2022-11-23T02:16:44.2077076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2077644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1rqmrt1n
2022-11-23T02:16:44.2078229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1rqmrt1n/_remote_module_non_scriptable.py
2022-11-23T02:16:44.2078817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.2079372Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.2079785Z ok (4.348s)
2022-11-23T02:16:44.2079968Z 
2022-11-23T02:16:44.2080273Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2080657Z Ran 1 test in 4.348s
2022-11-23T02:16:44.2080852Z 
2022-11-23T02:16:44.2080979Z OK
2022-11-23T02:16:44.2081129Z 
2022-11-23T02:16:44.2081286Z Generating XML reports...
2022-11-23T02:16:44.2081962Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015028.xml
2022-11-23T02:16:44.2082747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2083252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2083896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2084432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2084754Z 
2022-11-23T02:16:44.2084900Z Running tests...
2022-11-23T02:16:44.2085358Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2086200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2086768Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.2087360Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2087938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28696
2022-11-23T02:16:44.2088460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28697
2022-11-23T02:16:44.2089140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2089644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2090289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2090824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2091448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2091959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2092671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2093213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2093718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2094287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2095022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2095778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2096353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2096921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6rrwtkoh
2022-11-23T02:16:44.2097535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6rrwtkoh/_remote_module_non_scriptable.py
2022-11-23T02:16:44.2098100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2098670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvy96kzx
2022-11-23T02:16:44.2099272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvy96kzx/_remote_module_non_scriptable.py
2022-11-23T02:16:44.2099862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.2100397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.2100818Z ok (4.342s)
2022-11-23T02:16:44.2101002Z 
2022-11-23T02:16:44.2101304Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2101683Z Ran 1 test in 4.342s
2022-11-23T02:16:44.2101880Z 
2022-11-23T02:16:44.2102006Z OK
2022-11-23T02:16:44.2102175Z 
2022-11-23T02:16:44.2102328Z Generating XML reports...
2022-11-23T02:16:44.2103004Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015035.xml
2022-11-23T02:16:44.2103778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2104295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2105018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2105559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2105805Z 
2022-11-23T02:16:44.2105948Z Running tests...
2022-11-23T02:16:44.2106416Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2107011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2107541Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:16:44.2108100Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2108647Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28809
2022-11-23T02:16:44.2109166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28810
2022-11-23T02:16:44.2109832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2110347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2110988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2111512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2112155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2112723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2113373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2113889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2114415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2114986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2115698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2116511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2117105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2117652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2118205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjteq1w2c
2022-11-23T02:16:44.2118801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgdo6ao5_
2022-11-23T02:16:44.2119415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjteq1w2c/_remote_module_non_scriptable.py
2022-11-23T02:16:44.2120025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgdo6ao5_/_remote_module_non_scriptable.py
2022-11-23T02:16:44.2120457Z ok (4.245s)
2022-11-23T02:16:44.2120639Z 
2022-11-23T02:16:44.2120941Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2121339Z Ran 1 test in 4.245s
2022-11-23T02:16:44.2121535Z 
2022-11-23T02:16:44.2121645Z OK
2022-11-23T02:16:44.2121810Z 
2022-11-23T02:16:44.2121968Z Generating XML reports...
2022-11-23T02:16:44.2122646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015042.xml
2022-11-23T02:16:44.2123434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2123934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2124639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2125175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2125439Z 
2022-11-23T02:16:44.2125566Z Running tests...
2022-11-23T02:16:44.2126283Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2126886Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2127451Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2127988Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28922
2022-11-23T02:16:44.2128505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28923
2022-11-23T02:16:44.2129184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2129691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2130337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2130871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2131519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2132018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2132728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2133267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2133772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2134344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2135081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2135839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2136413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2136953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2137603Z STAGE:2022-11-23 01:50:53 28922:28922 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2138247Z STAGE:2022-11-23 01:50:53 28923:28923 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2139086Z STAGE:2022-11-23 01:50:53 28922:28922 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:50:53 28923:28923 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2139517Z 
2022-11-23T02:16:44.2140125Z STAGE:2022-11-23 01:50:53 28922:28922 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:50:53 28923:28923 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2140563Z 
2022-11-23T02:16:44.2140929Z STAGE:2022-11-23 01:50:53 28922:28922 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2141563Z STAGE:2022-11-23 01:50:53 28923:28923 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2142421Z STAGE:2022-11-23 01:50:53 28922:28922 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:50:53 28923:28923 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2142826Z 
2022-11-23T02:16:44.2143432Z STAGE:2022-11-23 01:50:53 28923:28923 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:50:53 28922:28922 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2143941Z 
2022-11-23T02:16:44.2144079Z ok (4.235s)
2022-11-23T02:16:44.2144262Z 
2022-11-23T02:16:44.2144561Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2144958Z Ran 1 test in 4.235s
2022-11-23T02:16:44.2145136Z 
2022-11-23T02:16:44.2145263Z OK
2022-11-23T02:16:44.2145430Z 
2022-11-23T02:16:44.2145587Z Generating XML reports...
2022-11-23T02:16:44.2146262Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015049.xml
2022-11-23T02:16:44.2147035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2147558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2148202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2148740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2149006Z 
2022-11-23T02:16:44.2149134Z Running tests...
2022-11-23T02:16:44.2149606Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2150207Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2150519Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2150774Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29035
2022-11-23T02:16:44.2151061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29036
2022-11-23T02:16:44.2151478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2151688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2152109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2152335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2152732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2152938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2153348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2153559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2153840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2154123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2154565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2154993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2155257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2155519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2155891Z STAGE:2022-11-23 01:50:59 29036:29036 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2156250Z STAGE:2022-11-23 01:50:59 29035:29035 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2157007Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2157210Z   warnings.warn(
2022-11-23T02:16:44.2157981Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2158128Z   warnings.warn(
2022-11-23T02:16:44.2158716Z STAGE:2022-11-23 01:50:59 29036:29036 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:50:59 29035:29035 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2158736Z 
2022-11-23T02:16:44.2159342Z STAGE:2022-11-23 01:50:59 29035:29035 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:50:59 29036:29036 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2159365Z 
2022-11-23T02:16:44.2159727Z STAGE:2022-11-23 01:50:59 29035:29035 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2160079Z STAGE:2022-11-23 01:50:59 29036:29036 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2160652Z STAGE:2022-11-23 01:50:59 29036:29036 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:50:59 29035:29035 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2160672Z 
2022-11-23T02:16:44.2161054Z STAGE:2022-11-23 01:50:59 29035:29035 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2161479Z STAGE:2022-11-23 01:50:59 29036:29036 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2161849Z STAGE:2022-11-23 01:50:59 29035:29035 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2162188Z STAGE:2022-11-23 01:50:59 29036:29036 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2162558Z STAGE:2022-11-23 01:50:59 29035:29035 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2162923Z STAGE:2022-11-23 01:50:59 29036:29036 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2163302Z STAGE:2022-11-23 01:50:59 29035:29035 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2163682Z STAGE:2022-11-23 01:50:59 29036:29036 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2163821Z ok (4.248s)
2022-11-23T02:16:44.2163840Z 
2022-11-23T02:16:44.2164140Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2164288Z Ran 1 test in 4.249s
2022-11-23T02:16:44.2164307Z 
2022-11-23T02:16:44.2164433Z OK
2022-11-23T02:16:44.2164452Z 
2022-11-23T02:16:44.2164595Z Generating XML reports...
2022-11-23T02:16:44.2165084Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015056.xml
2022-11-23T02:16:44.2165498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2165931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2166365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2166591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2166610Z 
2022-11-23T02:16:44.2166750Z Running tests...
2022-11-23T02:16:44.2167051Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2167383Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2167700Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2167955Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29148
2022-11-23T02:16:44.2168295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29149
2022-11-23T02:16:44.2168702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2168911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2169328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2169552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2169954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2170144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2170550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2170776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2171055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2171331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2171768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2172264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2172541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2172802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2173057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.2173338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.2173776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2174204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2174575Z STAGE:2022-11-23 01:51:06 29148:29148 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2175350Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2175501Z   warnings.warn(
2022-11-23T02:16:44.2175872Z STAGE:2022-11-23 01:51:06 29149:29149 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2176641Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2176786Z   warnings.warn(
2022-11-23T02:16:44.2177349Z STAGE:2022-11-23 01:51:06 29148:29148 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:51:06 29149:29149 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2177390Z 
2022-11-23T02:16:44.2177976Z STAGE:2022-11-23 01:51:06 29149:29149 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:51:06 29148:29148 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2178014Z 
2022-11-23T02:16:44.2178357Z STAGE:2022-11-23 01:51:06 29148:29148 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2178768Z STAGE:2022-11-23 01:51:06 29149:29149 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2179137Z STAGE:2022-11-23 01:51:06 29148:29148 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2179504Z STAGE:2022-11-23 01:51:06 29149:29149 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2179881Z STAGE:2022-11-23 01:51:06 29148:29148 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2180263Z STAGE:2022-11-23 01:51:06 29149:29149 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2180625Z STAGE:2022-11-23 01:51:06 29148:29148 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2180978Z STAGE:2022-11-23 01:51:06 29149:29149 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2181327Z STAGE:2022-11-23 01:51:06 29148:29148 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2181695Z STAGE:2022-11-23 01:51:06 29149:29149 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2182071Z STAGE:2022-11-23 01:51:06 29148:29148 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2182450Z STAGE:2022-11-23 01:51:06 29149:29149 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2182588Z ok (4.206s)
2022-11-23T02:16:44.2182608Z 
2022-11-23T02:16:44.2182911Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2183102Z Ran 1 test in 4.206s
2022-11-23T02:16:44.2183123Z 
2022-11-23T02:16:44.2183252Z OK
2022-11-23T02:16:44.2183271Z 
2022-11-23T02:16:44.2183431Z Generating XML reports...
2022-11-23T02:16:44.2183904Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015102.xml
2022-11-23T02:16:44.2184315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2184526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2184941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2185168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2185187Z 
2022-11-23T02:16:44.2185329Z Running tests...
2022-11-23T02:16:44.2185624Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2185976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2186269Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2186524Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29267
2022-11-23T02:16:44.2186781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29268
2022-11-23T02:16:44.2187191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2187399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2187817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2188041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2188441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2188649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2189039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2189314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2189596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2189873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2190313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2190747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2191015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2191279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2191471Z skip: Skipped due to small world size. (4.219s)
2022-11-23T02:16:44.2191490Z 
2022-11-23T02:16:44.2191773Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2191924Z Ran 1 test in 4.219s
2022-11-23T02:16:44.2191943Z 
2022-11-23T02:16:44.2192084Z OK (skipped=1)
2022-11-23T02:16:44.2192103Z 
2022-11-23T02:16:44.2192261Z Generating XML reports...
2022-11-23T02:16:44.2192746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015109.xml
2022-11-23T02:16:44.2193151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2193361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2193820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2194037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2194075Z 
2022-11-23T02:16:44.2194203Z Running tests...
2022-11-23T02:16:44.2194503Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2194854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2195161Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2195414Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29376
2022-11-23T02:16:44.2195666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29377
2022-11-23T02:16:44.2196073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2196282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2196679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2196904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2197306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2197516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2197924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2198144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2198424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2198705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2199138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2199555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2199873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2200134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2200506Z STAGE:2022-11-23 01:51:20 29376:29376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2200861Z STAGE:2022-11-23 01:51:20 29377:29377 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2201637Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2201786Z   warnings.warn(
2022-11-23T02:16:44.2202554Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2202704Z   warnings.warn(
2022-11-23T02:16:44.2203062Z STAGE:2022-11-23 01:51:20 29376:29376 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2203431Z STAGE:2022-11-23 01:51:20 29377:29377 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2203812Z STAGE:2022-11-23 01:51:20 29376:29376 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2204239Z STAGE:2022-11-23 01:51:20 29377:29377 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2204811Z STAGE:2022-11-23 01:51:20 29377:29377 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:51:20 29376:29376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2204835Z 
2022-11-23T02:16:44.2205202Z STAGE:2022-11-23 01:51:20 29376:29376 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2205562Z STAGE:2022-11-23 01:51:20 29377:29377 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2206153Z STAGE:2022-11-23 01:51:20 29376:29376 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2206538Z STAGE:2022-11-23 01:51:20 29377:29377 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2206899Z STAGE:2022-11-23 01:51:20 29376:29376 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2207240Z STAGE:2022-11-23 01:51:20 29377:29377 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2207821Z STAGE:2022-11-23 01:51:20 29376:29376 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:51:20 29377:29377 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2207844Z 
2022-11-23T02:16:44.2208443Z STAGE:2022-11-23 01:51:20 29376:29376 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:51:20 29377:29377 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2208463Z 
2022-11-23T02:16:44.2208590Z ok (4.208s)
2022-11-23T02:16:44.2208610Z 
2022-11-23T02:16:44.2208908Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2209055Z Ran 1 test in 4.208s
2022-11-23T02:16:44.2209073Z 
2022-11-23T02:16:44.2209199Z OK
2022-11-23T02:16:44.2209218Z 
2022-11-23T02:16:44.2209375Z Generating XML reports...
2022-11-23T02:16:44.2209865Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015116.xml
2022-11-23T02:16:44.2210274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2210470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2210968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2211193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2211213Z 
2022-11-23T02:16:44.2211357Z Running tests...
2022-11-23T02:16:44.2211651Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2211997Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2212316Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2212571Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29489
2022-11-23T02:16:44.2212823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29490
2022-11-23T02:16:44.2213214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2213428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2213843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2214068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2214467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2214677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2215140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2215374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2215637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2215919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2216398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2216833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2217096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2217358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2217731Z STAGE:2022-11-23 01:51:26 29489:29489 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2218091Z STAGE:2022-11-23 01:51:26 29490:29490 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2218864Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2219018Z   warnings.warn(
2022-11-23T02:16:44.2219767Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2219913Z   warnings.warn(
2022-11-23T02:16:44.2220502Z STAGE:2022-11-23 01:51:26 29489:29489 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:51:26 29490:29490 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2220522Z 
2022-11-23T02:16:44.2221127Z STAGE:2022-11-23 01:51:26 29489:29489 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:51:26 29490:29490 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2221199Z 
2022-11-23T02:16:44.2221338Z ok (4.272s)
2022-11-23T02:16:44.2221358Z 
2022-11-23T02:16:44.2221659Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2221806Z Ran 1 test in 4.272s
2022-11-23T02:16:44.2221825Z 
2022-11-23T02:16:44.2221951Z OK
2022-11-23T02:16:44.2221969Z 
2022-11-23T02:16:44.2222125Z Generating XML reports...
2022-11-23T02:16:44.2222610Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015123.xml
2022-11-23T02:16:44.2223001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2223216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2223637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2223869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2223889Z 
2022-11-23T02:16:44.2224031Z Running tests...
2022-11-23T02:16:44.2224327Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2224675Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2224971Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2225227Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29602
2022-11-23T02:16:44.2225511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29603
2022-11-23T02:16:44.2225925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2226138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2226560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2226785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2227180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2227385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2227792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2228004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2228285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2228562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2228999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2229428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2229692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2229954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2230324Z STAGE:2022-11-23 01:51:33 29602:29602 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2230686Z STAGE:2022-11-23 01:51:33 29603:29603 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2231250Z STAGE:2022-11-23 01:51:33 29602:29602 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:51:33 29603:29603 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2231346Z 
2022-11-23T02:16:44.2231937Z STAGE:2022-11-23 01:51:33 29602:29602 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:51:33 29603:29603 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2231975Z 
2022-11-23T02:16:44.2232320Z STAGE:2022-11-23 01:51:33 29602:29602 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2232674Z STAGE:2022-11-23 01:51:33 29603:29603 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2233251Z STAGE:2022-11-23 01:51:33 29602:29602 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:51:33 29603:29603 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2233272Z 
2022-11-23T02:16:44.2233653Z STAGE:2022-11-23 01:51:33 29602:29602 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2234035Z STAGE:2022-11-23 01:51:33 29603:29603 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2234180Z ok (4.236s)
2022-11-23T02:16:44.2234199Z 
2022-11-23T02:16:44.2234498Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2234644Z Ran 1 test in 4.237s
2022-11-23T02:16:44.2234663Z 
2022-11-23T02:16:44.2234787Z OK
2022-11-23T02:16:44.2234806Z 
2022-11-23T02:16:44.2234946Z Generating XML reports...
2022-11-23T02:16:44.2235433Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015129.xml
2022-11-23T02:16:44.2235883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2236102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2236515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2236740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2236763Z 
2022-11-23T02:16:44.2236904Z Running tests...
2022-11-23T02:16:44.2237202Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2237553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2237831Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-11-23T02:16:44.2237850Z 
2022-11-23T02:16:44.2238144Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2238291Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2238310Z 
2022-11-23T02:16:44.2238457Z OK (skipped=1)
2022-11-23T02:16:44.2238476Z 
2022-11-23T02:16:44.2238634Z Generating XML reports...
2022-11-23T02:16:44.2239118Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015136.xml
2022-11-23T02:16:44.2239528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2239743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2240159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2240368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2240387Z 
2022-11-23T02:16:44.2240532Z Running tests...
2022-11-23T02:16:44.2240829Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2241177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2241486Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-11-23T02:16:44.2241506Z 
2022-11-23T02:16:44.2241801Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2241998Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2242017Z 
2022-11-23T02:16:44.2242158Z OK (skipped=1)
2022-11-23T02:16:44.2242177Z 
2022-11-23T02:16:44.2242318Z Generating XML reports...
2022-11-23T02:16:44.2242804Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015139.xml
2022-11-23T02:16:44.2243209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2243418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2243834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2244059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2244079Z 
2022-11-23T02:16:44.2244224Z Running tests...
2022-11-23T02:16:44.2244523Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2244876Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2245155Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2245408Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29781
2022-11-23T02:16:44.2245657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29782
2022-11-23T02:16:44.2246257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2246537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2246964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2247190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2247587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2247782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2248187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2248410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2248689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2248966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2249406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2249837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2250100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2250369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2250626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.2250902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.2251334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2251763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2252133Z STAGE:2022-11-23 01:51:45 29782:29782 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2252492Z STAGE:2022-11-23 01:51:45 29781:29781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2252933Z STAGE:2022-11-23 01:51:45 29782:29782 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2253299Z STAGE:2022-11-23 01:51:45 29781:29781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2253680Z STAGE:2022-11-23 01:51:45 29782:29782 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2254041Z STAGE:2022-11-23 01:51:45 29781:29781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2254405Z STAGE:2022-11-23 01:51:45 29782:29782 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2254770Z STAGE:2022-11-23 01:51:45 29781:29781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2255139Z STAGE:2022-11-23 01:51:45 29781:29781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2255500Z STAGE:2022-11-23 01:51:45 29782:29782 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2255883Z STAGE:2022-11-23 01:51:45 29781:29781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2256261Z STAGE:2022-11-23 01:51:45 29782:29782 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2256400Z ok (4.368s)
2022-11-23T02:16:44.2256419Z 
2022-11-23T02:16:44.2256718Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2256850Z Ran 1 test in 4.368s
2022-11-23T02:16:44.2256869Z 
2022-11-23T02:16:44.2256995Z OK
2022-11-23T02:16:44.2257014Z 
2022-11-23T02:16:44.2257168Z Generating XML reports...
2022-11-23T02:16:44.2257703Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015141.xml
2022-11-23T02:16:44.2258118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2258328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2258750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2258978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2258998Z 
2022-11-23T02:16:44.2259142Z Running tests...
2022-11-23T02:16:44.2259420Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2259767Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2260060Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2260315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29900
2022-11-23T02:16:44.2260566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29901
2022-11-23T02:16:44.2260973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2261187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2261604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2261813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2262213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2262418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2262828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2263053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2263332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2263663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2264100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2264533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2264782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2265045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2265242Z skip: Skipped due to small world size. (4.355s)
2022-11-23T02:16:44.2265262Z 
2022-11-23T02:16:44.2265561Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2265709Z Ran 1 test in 4.355s
2022-11-23T02:16:44.2265728Z 
2022-11-23T02:16:44.2265870Z OK (skipped=1)
2022-11-23T02:16:44.2265892Z 
2022-11-23T02:16:44.2266048Z Generating XML reports...
2022-11-23T02:16:44.2266532Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015148.xml
2022-11-23T02:16:44.2266945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2267139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2267554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2267821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2267843Z 
2022-11-23T02:16:44.2267996Z Running tests...
2022-11-23T02:16:44.2268296Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2268644Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2268981Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-11-23T02:16:44.2269001Z 
2022-11-23T02:16:44.2269295Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2269441Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2269461Z 
2022-11-23T02:16:44.2269586Z OK (skipped=1)
2022-11-23T02:16:44.2269604Z 
2022-11-23T02:16:44.2269761Z Generating XML reports...
2022-11-23T02:16:44.2270243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015155.xml
2022-11-23T02:16:44.2270654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2270865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2271278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2271509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2271529Z 
2022-11-23T02:16:44.2271672Z Running tests...
2022-11-23T02:16:44.2271952Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2272300Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2272635Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-11-23T02:16:44.2272655Z 
2022-11-23T02:16:44.2272950Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2273095Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2273114Z 
2022-11-23T02:16:44.2273255Z OK (skipped=1)
2022-11-23T02:16:44.2273274Z 
2022-11-23T02:16:44.2273432Z Generating XML reports...
2022-11-23T02:16:44.2273914Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015157.xml
2022-11-23T02:16:44.2274376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2274570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2274984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2275208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2275228Z 
2022-11-23T02:16:44.2275365Z Running tests...
2022-11-23T02:16:44.2275662Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2276011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2276333Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-11-23T02:16:44.2276352Z 
2022-11-23T02:16:44.2276649Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2276794Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2276813Z 
2022-11-23T02:16:44.2276938Z OK (skipped=1)
2022-11-23T02:16:44.2276956Z 
2022-11-23T02:16:44.2277113Z Generating XML reports...
2022-11-23T02:16:44.2277592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015200.xml
2022-11-23T02:16:44.2277997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2278267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2278688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2278913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2278933Z 
2022-11-23T02:16:44.2279076Z Running tests...
2022-11-23T02:16:44.2279375Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2279703Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2280034Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-11-23T02:16:44.2280053Z 
2022-11-23T02:16:44.2280346Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2280489Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2280508Z 
2022-11-23T02:16:44.2280651Z OK (skipped=1)
2022-11-23T02:16:44.2280670Z 
2022-11-23T02:16:44.2280829Z Generating XML reports...
2022-11-23T02:16:44.2281311Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015202.xml
2022-11-23T02:16:44.2281714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2281910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2282323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2282547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2282566Z 
2022-11-23T02:16:44.2282708Z Running tests...
2022-11-23T02:16:44.2283007Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2283353Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2283662Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2283916Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30141
2022-11-23T02:16:44.2284169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30142
2022-11-23T02:16:44.2284614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2284828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2285244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2285470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2286095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2286313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2286727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2286950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2287214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2287500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2287936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2288367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2288632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2288966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2289110Z ok (4.381s)
2022-11-23T02:16:44.2289129Z 
2022-11-23T02:16:44.2289432Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2289580Z Ran 1 test in 4.382s
2022-11-23T02:16:44.2289600Z 
2022-11-23T02:16:44.2289710Z OK
2022-11-23T02:16:44.2289732Z 
2022-11-23T02:16:44.2289893Z Generating XML reports...
2022-11-23T02:16:44.2290384Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015204.xml
2022-11-23T02:16:44.2290788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2290998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2291412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2291641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2291661Z 
2022-11-23T02:16:44.2291806Z Running tests...
2022-11-23T02:16:44.2292102Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2292435Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2292746Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2292999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30250
2022-11-23T02:16:44.2293249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30251
2022-11-23T02:16:44.2293653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2293862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2294277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2294503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2294885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2295092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2295569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2295792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2296071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2296352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2296789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2297222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2297488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2297734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2298010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.2298285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.2298719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2299146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2299469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:16:44.2299747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:16:44.2300175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.2300607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.2300864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T02:16:44.2301138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T02:16:44.2301565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:16:44.2302001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:16:44.2302139Z ok (4.456s)
2022-11-23T02:16:44.2302159Z 
2022-11-23T02:16:44.2302456Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2302602Z Ran 1 test in 4.456s
2022-11-23T02:16:44.2302622Z 
2022-11-23T02:16:44.2302750Z OK
2022-11-23T02:16:44.2302769Z 
2022-11-23T02:16:44.2302927Z Generating XML reports...
2022-11-23T02:16:44.2303446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015211.xml
2022-11-23T02:16:44.2303857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2304069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2304482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2304711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2304730Z 
2022-11-23T02:16:44.2304873Z Running tests...
2022-11-23T02:16:44.2305166Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2305515Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2305879Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-11-23T02:16:44.2305899Z 
2022-11-23T02:16:44.2306175Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2306311Z Ran 1 test in 0.003s
2022-11-23T02:16:44.2306331Z 
2022-11-23T02:16:44.2306466Z OK (skipped=1)
2022-11-23T02:16:44.2306484Z 
2022-11-23T02:16:44.2306641Z Generating XML reports...
2022-11-23T02:16:44.2307127Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015218.xml
2022-11-23T02:16:44.2307533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2307743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2308157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2308370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2308407Z 
2022-11-23T02:16:44.2308534Z Running tests...
2022-11-23T02:16:44.2308830Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2309179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2309494Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2309747Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30416
2022-11-23T02:16:44.2310048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30417
2022-11-23T02:16:44.2310463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2310671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2311070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2311294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2311692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2311901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2312304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2312532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2312812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2313089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2313521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2313941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2314205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2314466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2314736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.2315014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.2315446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2315869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2316333Z STAGE:2022-11-23 01:52:25 30416:30416 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2317116Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2317249Z   warnings.warn(
2022-11-23T02:16:44.2317617Z STAGE:2022-11-23 01:52:25 30417:30417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2318392Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2318542Z   warnings.warn(
2022-11-23T02:16:44.2318920Z STAGE:2022-11-23 01:52:25 30417:30417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2319290Z STAGE:2022-11-23 01:52:25 30416:30416 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2319667Z STAGE:2022-11-23 01:52:25 30417:30417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2320049Z STAGE:2022-11-23 01:52:25 30416:30416 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2320409Z STAGE:2022-11-23 01:52:25 30417:30417 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2320793Z STAGE:2022-11-23 01:52:25 30416:30416 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2321170Z STAGE:2022-11-23 01:52:25 30417:30417 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2321532Z STAGE:2022-11-23 01:52:25 30416:30416 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2321918Z STAGE:2022-11-23 01:52:25 30417:30417 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2322297Z STAGE:2022-11-23 01:52:25 30416:30416 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2322435Z ok (4.342s)
2022-11-23T02:16:44.2322454Z 
2022-11-23T02:16:44.2322753Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2322900Z Ran 1 test in 4.342s
2022-11-23T02:16:44.2322920Z 
2022-11-23T02:16:44.2323044Z OK
2022-11-23T02:16:44.2323063Z 
2022-11-23T02:16:44.2323205Z Generating XML reports...
2022-11-23T02:16:44.2323696Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015221.xml
2022-11-23T02:16:44.2324109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2324322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2324751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2324960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2325001Z 
2022-11-23T02:16:44.2325130Z Running tests...
2022-11-23T02:16:44.2325425Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2325996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2326328Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2326585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30535
2022-11-23T02:16:44.2326835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30536
2022-11-23T02:16:44.2327243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2327530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2327930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2328157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2328557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2328762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2329172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2329397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2329677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2329961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2330384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2330812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2331074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2331336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2331668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.2331952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.2332383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2332814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2333182Z STAGE:2022-11-23 01:52:32 30536:30536 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2333522Z STAGE:2022-11-23 01:52:32 30535:30535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2334302Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2334452Z   warnings.warn(
2022-11-23T02:16:44.2335219Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2335367Z   warnings.warn(
2022-11-23T02:16:44.2335737Z STAGE:2022-11-23 01:52:32 30535:30535 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2336099Z STAGE:2022-11-23 01:52:32 30536:30536 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2336704Z STAGE:2022-11-23 01:52:32 30535:30535 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:52:32 30536:30536 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2336725Z 
2022-11-23T02:16:44.2337283Z STAGE:2022-11-23 01:52:32 30536:30536 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:52:32 30535:30535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2337303Z 
2022-11-23T02:16:44.2337665Z STAGE:2022-11-23 01:52:32 30536:30536 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2338085Z STAGE:2022-11-23 01:52:32 30535:30535 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2338462Z STAGE:2022-11-23 01:52:32 30536:30536 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2338823Z STAGE:2022-11-23 01:52:32 30535:30535 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2338947Z ok (4.218s)
2022-11-23T02:16:44.2338966Z 
2022-11-23T02:16:44.2339261Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2339404Z Ran 1 test in 4.218s
2022-11-23T02:16:44.2339424Z 
2022-11-23T02:16:44.2339547Z OK
2022-11-23T02:16:44.2339567Z 
2022-11-23T02:16:44.2339723Z Generating XML reports...
2022-11-23T02:16:44.2340212Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015228.xml
2022-11-23T02:16:44.2340622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2340815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2341231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2341454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2341474Z 
2022-11-23T02:16:44.2341613Z Running tests...
2022-11-23T02:16:44.2341911Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2342305Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2342636Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2342887Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30654
2022-11-23T02:16:44.2343141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30655
2022-11-23T02:16:44.2343531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2343740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2344149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2344369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2344765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2344969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2345376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2345594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2345857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2346133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2346562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2346988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2347253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2347513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2347779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.2348100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.2348529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2348943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2349310Z STAGE:2022-11-23 01:52:38 30655:30655 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2349662Z STAGE:2022-11-23 01:52:38 30654:30654 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2350440Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2350583Z   warnings.warn(
2022-11-23T02:16:44.2351356Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2351503Z   warnings.warn(
2022-11-23T02:16:44.2351871Z STAGE:2022-11-23 01:52:38 30654:30654 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2352233Z STAGE:2022-11-23 01:52:38 30655:30655 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2352656Z STAGE:2022-11-23 01:52:38 30654:30654 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2353027Z STAGE:2022-11-23 01:52:38 30655:30655 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2353384Z STAGE:2022-11-23 01:52:38 30654:30654 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2353740Z STAGE:2022-11-23 01:52:38 30655:30655 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2354098Z STAGE:2022-11-23 01:52:38 30654:30654 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2354461Z STAGE:2022-11-23 01:52:38 30655:30655 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2354832Z STAGE:2022-11-23 01:52:38 30654:30654 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2355205Z STAGE:2022-11-23 01:52:38 30655:30655 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2355342Z ok (4.256s)
2022-11-23T02:16:44.2355363Z 
2022-11-23T02:16:44.2355656Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2355788Z Ran 1 test in 4.256s
2022-11-23T02:16:44.2355807Z 
2022-11-23T02:16:44.2355928Z OK
2022-11-23T02:16:44.2355947Z 
2022-11-23T02:16:44.2356101Z Generating XML reports...
2022-11-23T02:16:44.2356645Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015234.xml
2022-11-23T02:16:44.2357044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2357246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2357655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2357876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2357900Z 
2022-11-23T02:16:44.2358042Z Running tests...
2022-11-23T02:16:44.2358324Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2358668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2358980Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2359288Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30773
2022-11-23T02:16:44.2359534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30774
2022-11-23T02:16:44.2359937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2360138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2360551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2360757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2361153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2361356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2361765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2361980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2362254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2362527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2362960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2363436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2363692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2363949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2364220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.2364488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.2364912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2365334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2365914Z STAGE:2022-11-23 01:52:45 30773:30773 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2366285Z STAGE:2022-11-23 01:52:45 30774:30774 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2367051Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2367189Z   warnings.warn(
2022-11-23T02:16:44.2367949Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2368093Z   warnings.warn(
2022-11-23T02:16:44.2368677Z STAGE:2022-11-23 01:52:45 30773:30773 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:52:45 30774:30774 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2368697Z 
2022-11-23T02:16:44.2369290Z STAGE:2022-11-23 01:52:45 30773:30773 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:52:45 30774:30774 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2369382Z 
2022-11-23T02:16:44.2369746Z STAGE:2022-11-23 01:52:45 30773:30773 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2370095Z STAGE:2022-11-23 01:52:45 30774:30774 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2370455Z STAGE:2022-11-23 01:52:45 30774:30774 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2370814Z STAGE:2022-11-23 01:52:45 30773:30773 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2371187Z STAGE:2022-11-23 01:52:45 30774:30774 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2371565Z STAGE:2022-11-23 01:52:45 30773:30773 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2371685Z ok (4.261s)
2022-11-23T02:16:44.2371720Z 
2022-11-23T02:16:44.2372001Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2372149Z Ran 1 test in 4.261s
2022-11-23T02:16:44.2372169Z 
2022-11-23T02:16:44.2372290Z OK
2022-11-23T02:16:44.2372309Z 
2022-11-23T02:16:44.2372459Z Generating XML reports...
2022-11-23T02:16:44.2372943Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015241.xml
2022-11-23T02:16:44.2373350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2373559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2374037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2374257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2374276Z 
2022-11-23T02:16:44.2374417Z Running tests...
2022-11-23T02:16:44.2374714Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2375063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2375371Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2375621Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30892
2022-11-23T02:16:44.2375870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30893
2022-11-23T02:16:44.2376273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2376469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2376878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2377097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2377489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2377697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2378106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2378319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2378598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2378869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2379290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2379721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2380050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2380309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2380502Z skip: Skipped due to small world size. (4.239s)
2022-11-23T02:16:44.2380522Z 
2022-11-23T02:16:44.2380822Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2380968Z Ran 1 test in 4.239s
2022-11-23T02:16:44.2380987Z 
2022-11-23T02:16:44.2381125Z OK (skipped=1)
2022-11-23T02:16:44.2381144Z 
2022-11-23T02:16:44.2381295Z Generating XML reports...
2022-11-23T02:16:44.2381768Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015248.xml
2022-11-23T02:16:44.2382167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2382374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2382784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2383002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2383021Z 
2022-11-23T02:16:44.2383157Z Running tests...
2022-11-23T02:16:44.2383454Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2383800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2384097Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2384393Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31001
2022-11-23T02:16:44.2384649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31002
2022-11-23T02:16:44.2385057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2385264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2385677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2385896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2386288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2386489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2386890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2387108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2387383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2387652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2388086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2388514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2388774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2389033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2389212Z skip: Skipped due to small world size. (4.222s)
2022-11-23T02:16:44.2389245Z 
2022-11-23T02:16:44.2389531Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2389672Z Ran 1 test in 4.222s
2022-11-23T02:16:44.2389691Z 
2022-11-23T02:16:44.2389828Z OK (skipped=1)
2022-11-23T02:16:44.2389846Z 
2022-11-23T02:16:44.2390054Z Generating XML reports...
2022-11-23T02:16:44.2390538Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015255.xml
2022-11-23T02:16:44.2390938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2391139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2391545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2391755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2391791Z 
2022-11-23T02:16:44.2391919Z Running tests...
2022-11-23T02:16:44.2392210Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2392553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2392873Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2393129Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31110
2022-11-23T02:16:44.2393376Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31111
2022-11-23T02:16:44.2393779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2393987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2394427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2394656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2395059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2395262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2395666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2395885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2396160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2396430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2396850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2397283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2397541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2397798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2397989Z skip: Skipped due to small world size. (4.205s)
2022-11-23T02:16:44.2398009Z 
2022-11-23T02:16:44.2398302Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2398445Z Ran 1 test in 4.205s
2022-11-23T02:16:44.2398464Z 
2022-11-23T02:16:44.2398600Z OK (skipped=1)
2022-11-23T02:16:44.2398619Z 
2022-11-23T02:16:44.2398771Z Generating XML reports...
2022-11-23T02:16:44.2399241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015302.xml
2022-11-23T02:16:44.2399648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2399854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2400263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2400537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2400556Z 
2022-11-23T02:16:44.2400693Z Running tests...
2022-11-23T02:16:44.2400989Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2401333Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2401645Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2401883Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31219
2022-11-23T02:16:44.2402135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31220
2022-11-23T02:16:44.2402539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2402745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2403151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2403375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2403765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2403968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2404360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2404623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2404904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2405171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2405605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2406248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2406511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2406765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2406953Z skip: Skipped due to small world size. (4.252s)
2022-11-23T02:16:44.2406972Z 
2022-11-23T02:16:44.2407254Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2407400Z Ran 1 test in 4.252s
2022-11-23T02:16:44.2407420Z 
2022-11-23T02:16:44.2407559Z OK (skipped=1)
2022-11-23T02:16:44.2407578Z 
2022-11-23T02:16:44.2407731Z Generating XML reports...
2022-11-23T02:16:44.2408217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015309.xml
2022-11-23T02:16:44.2408623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2408830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2409233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2409453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2409473Z 
2022-11-23T02:16:44.2409599Z Running tests...
2022-11-23T02:16:44.2409892Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2410235Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2410530Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2410778Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31328
2022-11-23T02:16:44.2411103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31329
2022-11-23T02:16:44.2411506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2411709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2412101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2412320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2412718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2412917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2413325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2413544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2413818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2414086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2414516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2414932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2415251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2415519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2415893Z STAGE:2022-11-23 01:53:19 31328:31328 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2416286Z STAGE:2022-11-23 01:53:19 31329:31329 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2417065Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2417208Z   warnings.warn(
2022-11-23T02:16:44.2417982Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2418119Z   warnings.warn(
2022-11-23T02:16:44.2418703Z STAGE:2022-11-23 01:53:19 31328:31328 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:53:19 31329:31329 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2418728Z 
2022-11-23T02:16:44.2419330Z STAGE:2022-11-23 01:53:19 31328:31328 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:53:19 31329:31329 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2419351Z 
2022-11-23T02:16:44.2419694Z STAGE:2022-11-23 01:53:19 31328:31328 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2420039Z STAGE:2022-11-23 01:53:19 31329:31329 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2420406Z STAGE:2022-11-23 01:53:19 31328:31328 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2420773Z STAGE:2022-11-23 01:53:19 31329:31329 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2421149Z STAGE:2022-11-23 01:53:19 31328:31328 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2421585Z STAGE:2022-11-23 01:53:19 31329:31329 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2421719Z ok (4.254s)
2022-11-23T02:16:44.2421738Z 
2022-11-23T02:16:44.2422035Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2422178Z Ran 1 test in 4.254s
2022-11-23T02:16:44.2422197Z 
2022-11-23T02:16:44.2422307Z OK
2022-11-23T02:16:44.2422325Z 
2022-11-23T02:16:44.2422478Z Generating XML reports...
2022-11-23T02:16:44.2422961Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015315.xml
2022-11-23T02:16:44.2423366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2423571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2423983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2424204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2424224Z 
2022-11-23T02:16:44.2424362Z Running tests...
2022-11-23T02:16:44.2424642Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2424984Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2425314Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2425559Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31441
2022-11-23T02:16:44.2425850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31442
2022-11-23T02:16:44.2426257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2426460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2426877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2427099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2427482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2427685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2428087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2428310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2428588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2428863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2429299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2429727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2429991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2430237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2431018Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2431165Z   warnings.warn(
2022-11-23T02:16:44.2431929Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2432127Z   warnings.warn(
2022-11-23T02:16:44.2432260Z ok (4.145s)
2022-11-23T02:16:44.2432280Z 
2022-11-23T02:16:44.2432576Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2432721Z Ran 1 test in 4.145s
2022-11-23T02:16:44.2432741Z 
2022-11-23T02:16:44.2432865Z OK
2022-11-23T02:16:44.2432884Z 
2022-11-23T02:16:44.2433025Z Generating XML reports...
2022-11-23T02:16:44.2433511Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015322.xml
2022-11-23T02:16:44.2433912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2434119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2434535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2434755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2434774Z 
2022-11-23T02:16:44.2434910Z Running tests...
2022-11-23T02:16:44.2435200Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2435548Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2435834Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2436165Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31550
2022-11-23T02:16:44.2436426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31551
2022-11-23T02:16:44.2436824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2437032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2437435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2437655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2438045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2438234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2438646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2438866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2439143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2439415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2439851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2440287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2440544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2440805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2441160Z STAGE:2022-11-23 01:53:33 31550:31550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2441512Z STAGE:2022-11-23 01:53:33 31551:31551 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2442279Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2442475Z   warnings.warn(
2022-11-23T02:16:44.2443242Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2443383Z   warnings.warn(
2022-11-23T02:16:44.2443756Z STAGE:2022-11-23 01:53:33 31550:31550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2444120Z STAGE:2022-11-23 01:53:33 31551:31551 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2444502Z STAGE:2022-11-23 01:53:33 31550:31550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2444878Z STAGE:2022-11-23 01:53:33 31551:31551 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2445230Z STAGE:2022-11-23 01:53:33 31550:31550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2445581Z STAGE:2022-11-23 01:53:33 31551:31551 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2446252Z STAGE:2022-11-23 01:53:33 31550:31550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2446626Z STAGE:2022-11-23 01:53:33 31551:31551 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2447070Z STAGE:2022-11-23 01:53:33 31550:31550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2447459Z STAGE:2022-11-23 01:53:33 31551:31551 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2447591Z ok (4.238s)
2022-11-23T02:16:44.2447611Z 
2022-11-23T02:16:44.2447910Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2448045Z Ran 1 test in 4.239s
2022-11-23T02:16:44.2448077Z 
2022-11-23T02:16:44.2448186Z OK
2022-11-23T02:16:44.2448205Z 
2022-11-23T02:16:44.2448360Z Generating XML reports...
2022-11-23T02:16:44.2448844Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015329.xml
2022-11-23T02:16:44.2449247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2449451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2449869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2450092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2450111Z 
2022-11-23T02:16:44.2450253Z Running tests...
2022-11-23T02:16:44.2450536Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2450886Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2451195Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2451442Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31663
2022-11-23T02:16:44.2451694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31664
2022-11-23T02:16:44.2452099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2452307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2452718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2452942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2453323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2453615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2454018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2454239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2454515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2454794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2455227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2455653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2455905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2456278Z STAGE:2022-11-23 01:53:39 31664:31664 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2457052Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2457195Z   warnings.warn(
2022-11-23T02:16:44.2457502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2457876Z STAGE:2022-11-23 01:53:39 31663:31663 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2458636Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2458781Z   warnings.warn(
2022-11-23T02:16:44.2459361Z STAGE:2022-11-23 01:53:39 31664:31664 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:53:39 31663:31663 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2459382Z 
2022-11-23T02:16:44.2459980Z STAGE:2022-11-23 01:53:39 31664:31664 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:53:39 31663:31663 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2460003Z 
2022-11-23T02:16:44.2460364Z STAGE:2022-11-23 01:53:39 31664:31664 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2460713Z STAGE:2022-11-23 01:53:39 31663:31663 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2461060Z STAGE:2022-11-23 01:53:39 31663:31663 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2461426Z STAGE:2022-11-23 01:53:39 31664:31664 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2461802Z STAGE:2022-11-23 01:53:39 31663:31663 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2462174Z STAGE:2022-11-23 01:53:39 31664:31664 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2462309Z ok (4.306s)
2022-11-23T02:16:44.2462328Z 
2022-11-23T02:16:44.2462619Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2462767Z Ran 1 test in 4.306s
2022-11-23T02:16:44.2462787Z 
2022-11-23T02:16:44.2462907Z OK
2022-11-23T02:16:44.2462925Z 
2022-11-23T02:16:44.2463066Z Generating XML reports...
2022-11-23T02:16:44.2463554Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015336.xml
2022-11-23T02:16:44.2464018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2464222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2464634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2464855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2464874Z 
2022-11-23T02:16:44.2465015Z Running tests...
2022-11-23T02:16:44.2465308Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2465660Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2465949Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2466201Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31776
2022-11-23T02:16:44.2466451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31777
2022-11-23T02:16:44.2466853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2467060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2467469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2467692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2468139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2468335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2468741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2468959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2469238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2469510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2469942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2470365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2470632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2470997Z STAGE:2022-11-23 01:53:46 31777:31777 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2471768Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2471904Z   warnings.warn(
2022-11-23T02:16:44.2472166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2472533Z STAGE:2022-11-23 01:53:46 31776:31776 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2473300Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:16:44.2473447Z   warnings.warn(
2022-11-23T02:16:44.2474024Z STAGE:2022-11-23 01:53:46 31776:31776 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:53:46 31777:31777 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2474091Z 
2022-11-23T02:16:44.2474696Z STAGE:2022-11-23 01:53:46 31777:31777 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:53:46 31776:31776 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2474716Z 
2022-11-23T02:16:44.2475076Z STAGE:2022-11-23 01:53:46 31777:31777 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2475423Z STAGE:2022-11-23 01:53:46 31776:31776 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2475789Z STAGE:2022-11-23 01:53:46 31777:31777 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2476149Z STAGE:2022-11-23 01:53:46 31776:31776 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2476510Z STAGE:2022-11-23 01:53:46 31777:31777 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2476888Z STAGE:2022-11-23 01:53:46 31776:31776 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2477024Z ok (4.316s)
2022-11-23T02:16:44.2477043Z 
2022-11-23T02:16:44.2477343Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2477487Z Ran 1 test in 4.317s
2022-11-23T02:16:44.2477506Z 
2022-11-23T02:16:44.2477628Z OK
2022-11-23T02:16:44.2477647Z 
2022-11-23T02:16:44.2477804Z Generating XML reports...
2022-11-23T02:16:44.2478287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015342.xml
2022-11-23T02:16:44.2478723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2478942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2479354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2479581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2479601Z 
2022-11-23T02:16:44.2479732Z Running tests...
2022-11-23T02:16:44.2480025Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2480368Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2480689Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2480941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31889
2022-11-23T02:16:44.2481181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31890
2022-11-23T02:16:44.2481586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2481794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2482204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2482430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2482826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2483031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2483439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2483646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2483929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2484203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2484656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2485138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2485400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2485657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2485970Z ok (4.239s)
2022-11-23T02:16:44.2485990Z 
2022-11-23T02:16:44.2486292Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2486423Z Ran 1 test in 4.240s
2022-11-23T02:16:44.2486450Z 
2022-11-23T02:16:44.2486575Z OK
2022-11-23T02:16:44.2486594Z 
2022-11-23T02:16:44.2486750Z Generating XML reports...
2022-11-23T02:16:44.2487236Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015349.xml
2022-11-23T02:16:44.2487635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2487846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2488252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2488476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2488495Z 
2022-11-23T02:16:44.2488626Z Running tests...
2022-11-23T02:16:44.2488905Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2489325Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2489638Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2489887Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31998
2022-11-23T02:16:44.2490139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31999
2022-11-23T02:16:44.2490543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2490748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2491156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2491364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2491758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2491961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2492363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2492586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2492864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2493137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2493569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2493995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2494246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2494506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2494776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.2495046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.2495548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2495974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2496339Z STAGE:2022-11-23 01:54:00 31998:31998 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2496693Z STAGE:2022-11-23 01:54:00 31999:31999 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2497063Z STAGE:2022-11-23 01:54:00 31998:31998 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2497409Z STAGE:2022-11-23 01:54:00 31999:31999 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2497788Z STAGE:2022-11-23 01:54:00 31998:31998 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2498167Z STAGE:2022-11-23 01:54:00 31999:31999 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2498526Z STAGE:2022-11-23 01:54:00 31998:31998 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2498872Z STAGE:2022-11-23 01:54:00 31999:31999 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2499234Z STAGE:2022-11-23 01:54:00 31998:31998 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2499867Z STAGE:2022-11-23 01:54:00 31999:31999 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:54:00 31998:31998 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2499889Z 
2022-11-23T02:16:44.2500267Z STAGE:2022-11-23 01:54:00 31999:31999 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2500401Z ok (4.236s)
2022-11-23T02:16:44.2500420Z 
2022-11-23T02:16:44.2500707Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2500840Z Ran 1 test in 4.236s
2022-11-23T02:16:44.2500860Z 
2022-11-23T02:16:44.2500987Z OK
2022-11-23T02:16:44.2501005Z 
2022-11-23T02:16:44.2501156Z Generating XML reports...
2022-11-23T02:16:44.2501641Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015356.xml
2022-11-23T02:16:44.2502046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2502254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2502665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2502892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2502911Z 
2022-11-23T02:16:44.2503038Z Running tests...
2022-11-23T02:16:44.2503340Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2503687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2503985Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2504231Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32117
2022-11-23T02:16:44.2504480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32118
2022-11-23T02:16:44.2504881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2505088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2505484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2505706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2506162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2506363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2506766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2506983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2507259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2507534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2507967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2508382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2508654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2508914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2509181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.2509453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.2509927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2510362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2510725Z STAGE:2022-11-23 01:54:07 32117:32117 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2511080Z STAGE:2022-11-23 01:54:07 32118:32118 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2511436Z STAGE:2022-11-23 01:54:07 32117:32117 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2511792Z STAGE:2022-11-23 01:54:07 32118:32118 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2512171Z STAGE:2022-11-23 01:54:07 32117:32117 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2512547Z STAGE:2022-11-23 01:54:07 32118:32118 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2512904Z STAGE:2022-11-23 01:54:07 32117:32117 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2513258Z STAGE:2022-11-23 01:54:07 32118:32118 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2513825Z STAGE:2022-11-23 01:54:07 32118:32118 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:54:07 32117:32117 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2513849Z 
2022-11-23T02:16:44.2514450Z STAGE:2022-11-23 01:54:07 32117:32117 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:54:07 32118:32118 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2514469Z 
2022-11-23T02:16:44.2514604Z ok (4.240s)
2022-11-23T02:16:44.2514623Z 
2022-11-23T02:16:44.2514919Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2515052Z Ran 1 test in 4.240s
2022-11-23T02:16:44.2515082Z 
2022-11-23T02:16:44.2515190Z OK
2022-11-23T02:16:44.2515212Z 
2022-11-23T02:16:44.2515369Z Generating XML reports...
2022-11-23T02:16:44.2515851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015403.xml
2022-11-23T02:16:44.2516296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2516562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2516978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2517200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2517219Z 
2022-11-23T02:16:44.2517358Z Running tests...
2022-11-23T02:16:44.2517637Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2517983Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2518293Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2518542Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32236
2022-11-23T02:16:44.2518789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32237
2022-11-23T02:16:44.2519197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2519404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2519816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2520040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2520424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2520674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2521088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2521305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2521583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2521860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2522294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2522717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2522963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2523228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2523499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.2523772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.2524212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2524637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2525005Z STAGE:2022-11-23 01:54:13 32236:32236 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2525356Z STAGE:2022-11-23 01:54:13 32237:32237 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2525946Z STAGE:2022-11-23 01:54:13 32236:32236 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2526308Z STAGE:2022-11-23 01:54:13 32237:32237 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2526691Z STAGE:2022-11-23 01:54:13 32236:32236 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2527070Z STAGE:2022-11-23 01:54:13 32237:32237 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2527525Z STAGE:2022-11-23 01:54:13 32237:32237 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2527885Z STAGE:2022-11-23 01:54:13 32236:32236 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2528247Z STAGE:2022-11-23 01:54:13 32236:32236 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2528605Z STAGE:2022-11-23 01:54:13 32237:32237 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2528984Z STAGE:2022-11-23 01:54:13 32236:32236 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2529361Z STAGE:2022-11-23 01:54:13 32237:32237 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2529484Z ok (4.217s)
2022-11-23T02:16:44.2529519Z 
2022-11-23T02:16:44.2529804Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2529956Z Ran 1 test in 4.217s
2022-11-23T02:16:44.2529975Z 
2022-11-23T02:16:44.2530098Z OK
2022-11-23T02:16:44.2530117Z 
2022-11-23T02:16:44.2530272Z Generating XML reports...
2022-11-23T02:16:44.2530758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015410.xml
2022-11-23T02:16:44.2531162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2531427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2531945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2532164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2532183Z 
2022-11-23T02:16:44.2532357Z Running tests...
2022-11-23T02:16:44.2532692Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2533206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2533550Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2533890Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32355
2022-11-23T02:16:44.2534182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32356
2022-11-23T02:16:44.2534577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2534828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2535280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2535544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2535980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2536270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2536735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2536999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2537264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2537579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2538056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2538528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2538884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2539182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2539540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.2539886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.2540360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2540830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.2541187Z STAGE:2022-11-23 01:54:20 32355:32355 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2541580Z STAGE:2022-11-23 01:54:20 32356:32356 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2541991Z STAGE:2022-11-23 01:54:20 32355:32355 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2542388Z STAGE:2022-11-23 01:54:20 32356:32356 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2542813Z STAGE:2022-11-23 01:54:20 32355:32355 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2543265Z STAGE:2022-11-23 01:54:20 32356:32356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2543911Z STAGE:2022-11-23 01:54:20 32356:32356 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:54:20 32355:32355 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2543933Z 
2022-11-23T02:16:44.2544345Z STAGE:2022-11-23 01:54:20 32355:32355 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2544755Z STAGE:2022-11-23 01:54:20 32356:32356 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2545122Z STAGE:2022-11-23 01:54:20 32355:32355 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2545542Z STAGE:2022-11-23 01:54:20 32356:32356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2545727Z ok (4.338s)
2022-11-23T02:16:44.2545747Z 
2022-11-23T02:16:44.2546080Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2546256Z Ran 1 test in 4.338s
2022-11-23T02:16:44.2546277Z 
2022-11-23T02:16:44.2546501Z OK
2022-11-23T02:16:44.2546521Z 
2022-11-23T02:16:44.2546711Z Generating XML reports...
2022-11-23T02:16:44.2547238Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015416.xml
2022-11-23T02:16:44.2547683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2547874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2548338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2548601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2548621Z 
2022-11-23T02:16:44.2548796Z Running tests...
2022-11-23T02:16:44.2549132Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2549557Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2549895Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2550186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32474
2022-11-23T02:16:44.2550420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32475
2022-11-23T02:16:44.2550871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2551173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2551628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2551885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2552358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2552637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2553087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2553356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2553621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2553938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2554417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2554887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2555184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2555479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2555797Z skip: Skipped due to small world size. (4.112s)
2022-11-23T02:16:44.2555818Z 
2022-11-23T02:16:44.2556162Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2556341Z Ran 1 test in 4.112s
2022-11-23T02:16:44.2556361Z 
2022-11-23T02:16:44.2556487Z OK (skipped=1)
2022-11-23T02:16:44.2556509Z 
2022-11-23T02:16:44.2556697Z Generating XML reports...
2022-11-23T02:16:44.2557221Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015423.xml
2022-11-23T02:16:44.2557665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2557944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2558405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2558706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2558727Z 
2022-11-23T02:16:44.2558905Z Running tests...
2022-11-23T02:16:44.2559190Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2559574Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2559908Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2560193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32583
2022-11-23T02:16:44.2560477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32584
2022-11-23T02:16:44.2560931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2561174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2561667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2561927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2562316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2562557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2563057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2563312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2563674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2563988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2564466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2564972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2565270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2565514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2565970Z skip: Skipped due to small world size. (4.318s)
2022-11-23T02:16:44.2565991Z 
2022-11-23T02:16:44.2566347Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2566543Z Ran 1 test in 4.318s
2022-11-23T02:16:44.2566563Z 
2022-11-23T02:16:44.2566739Z OK (skipped=1)
2022-11-23T02:16:44.2566758Z 
2022-11-23T02:16:44.2566943Z Generating XML reports...
2022-11-23T02:16:44.2567466Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015430.xml
2022-11-23T02:16:44.2568033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2568239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2568701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2568976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2568996Z 
2022-11-23T02:16:44.2569164Z Running tests...
2022-11-23T02:16:44.2569533Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2569918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2570258Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2570550Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32692
2022-11-23T02:16:44.2570879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32693
2022-11-23T02:16:44.2571275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2571529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2571982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2572245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2572680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2572922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2573367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2573630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2573894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2574250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2574723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2575262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2575599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2575896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2576126Z skip: Skipped due to small world size. (4.152s)
2022-11-23T02:16:44.2576147Z 
2022-11-23T02:16:44.2576487Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2576685Z Ran 1 test in 4.152s
2022-11-23T02:16:44.2576705Z 
2022-11-23T02:16:44.2576833Z OK (skipped=1)
2022-11-23T02:16:44.2576852Z 
2022-11-23T02:16:44.2577083Z Generating XML reports...
2022-11-23T02:16:44.2577610Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015437.xml
2022-11-23T02:16:44.2578058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2578303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2578752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2579013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2579032Z 
2022-11-23T02:16:44.2579222Z Running tests...
2022-11-23T02:16:44.2579605Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2579944Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2580319Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2580611Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32801
2022-11-23T02:16:44.2580937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32802
2022-11-23T02:16:44.2581382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2581638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2582091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2582354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2582741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2582988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2583467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2583730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2584049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2584373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2584846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2585316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2585619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2585864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2586089Z skip: Skipped due to small world size. (4.149s)
2022-11-23T02:16:44.2586109Z 
2022-11-23T02:16:44.2586542Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2586767Z Ran 1 test in 4.150s
2022-11-23T02:16:44.2586787Z 
2022-11-23T02:16:44.2586975Z OK (skipped=1)
2022-11-23T02:16:44.2586994Z 
2022-11-23T02:16:44.2587184Z Generating XML reports...
2022-11-23T02:16:44.2587712Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015443.xml
2022-11-23T02:16:44.2588152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2588399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2588799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2589057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2589077Z 
2022-11-23T02:16:44.2589291Z Running tests...
2022-11-23T02:16:44.2589639Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2590021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2590339Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2590626Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32910
2022-11-23T02:16:44.2590908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32911
2022-11-23T02:16:44.2591349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2591597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2592058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2592397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2592839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2593076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2593523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2593778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2594090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2594354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2594833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2595299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2595639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2595936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2596345Z STAGE:2022-11-23 01:54:54 32911:32911 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2596741Z STAGE:2022-11-23 01:54:54 32910:32910 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2597150Z STAGE:2022-11-23 01:54:54 32910:32910 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2597569Z STAGE:2022-11-23 01:54:54 32911:32911 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2598164Z STAGE:2022-11-23 01:54:54 32910:32910 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:54:54 32911:32911 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2598289Z 
2022-11-23T02:16:44.2598644Z STAGE:2022-11-23 01:54:54 32910:32910 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2599071Z STAGE:2022-11-23 01:54:54 32911:32911 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2599723Z STAGE:2022-11-23 01:54:54 32911:32911 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:54:54 32910:32910 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2599743Z 
2022-11-23T02:16:44.2600387Z STAGE:2022-11-23 01:54:54 32911:32911 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:54:54 32910:32910 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2600407Z 
2022-11-23T02:16:44.2600577Z ok (4.224s)
2022-11-23T02:16:44.2600597Z 
2022-11-23T02:16:44.2600934Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2601132Z Ran 1 test in 4.224s
2022-11-23T02:16:44.2601152Z 
2022-11-23T02:16:44.2601313Z OK
2022-11-23T02:16:44.2601332Z 
2022-11-23T02:16:44.2601526Z Generating XML reports...
2022-11-23T02:16:44.2602000Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015450.xml
2022-11-23T02:16:44.2602443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2602729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2603279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2603563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2603582Z 
2022-11-23T02:16:44.2603762Z Running tests...
2022-11-23T02:16:44.2604096Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2604484Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2604830Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2605065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33023
2022-11-23T02:16:44.2605352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33024
2022-11-23T02:16:44.2606067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2606339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2606803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2607062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2607497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2607744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2608134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2608391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2608712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2609079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2609560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2610026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2610327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2610823Z STAGE:2022-11-23 01:55:01 33024:33024 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2611152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2611495Z STAGE:2022-11-23 01:55:01 33023:33023 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2612114Z STAGE:2022-11-23 01:55:01 33023:33023 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:55:01 33024:33024 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2612138Z 
2022-11-23T02:16:44.2612795Z STAGE:2022-11-23 01:55:01 33024:33024 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:55:01 33023:33023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2612815Z 
2022-11-23T02:16:44.2613454Z STAGE:2022-11-23 01:55:01 33023:33023 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:55:01 33024:33024 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2613480Z 
2022-11-23T02:16:44.2613887Z STAGE:2022-11-23 01:55:01 33023:33023 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2614509Z STAGE:2022-11-23 01:55:01 33024:33024 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:55:01 33023:33023 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2614529Z 
2022-11-23T02:16:44.2615005Z STAGE:2022-11-23 01:55:01 33024:33024 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2615186Z ok (4.298s)
2022-11-23T02:16:44.2615206Z 
2022-11-23T02:16:44.2615550Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2615769Z Ran 1 test in 4.298s
2022-11-23T02:16:44.2615789Z 
2022-11-23T02:16:44.2616000Z OK
2022-11-23T02:16:44.2616020Z 
2022-11-23T02:16:44.2616289Z Generating XML reports...
2022-11-23T02:16:44.2616766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015457.xml
2022-11-23T02:16:44.2617210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2617450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2618009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2618274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2618295Z 
2022-11-23T02:16:44.2618474Z Running tests...
2022-11-23T02:16:44.2618808Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2619190Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2619563Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2619804Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33136
2022-11-23T02:16:44.2620091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33137
2022-11-23T02:16:44.2620543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2620783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2621237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2621498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2621938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2622258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2622662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2622959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2623283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2623630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2624108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2624553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2639377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2639653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2640065Z STAGE:2022-11-23 01:55:08 33137:33137 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2640880Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:16:44.2641031Z   warnings.warn(
2022-11-23T02:16:44.2641494Z STAGE:2022-11-23 01:55:08 33136:33136 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2642313Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:16:44.2642464Z   warnings.warn(
2022-11-23T02:16:44.2642836Z STAGE:2022-11-23 01:55:09 33136:33136 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2643186Z STAGE:2022-11-23 01:55:09 33137:33137 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2643568Z STAGE:2022-11-23 01:55:09 33136:33136 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2643945Z STAGE:2022-11-23 01:55:09 33137:33137 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2644308Z STAGE:2022-11-23 01:55:09 33136:33136 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2644660Z STAGE:2022-11-23 01:55:09 33137:33137 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2645238Z STAGE:2022-11-23 01:55:09 33137:33137 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:55:09 33136:33136 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2645264Z 
2022-11-23T02:16:44.2646193Z STAGE:2022-11-23 01:55:09 33137:33137 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:55:09 33136:33136 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2646217Z 
2022-11-23T02:16:44.2646360Z ok (5.350s)
2022-11-23T02:16:44.2646380Z 
2022-11-23T02:16:44.2646686Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2646838Z Ran 1 test in 5.350s
2022-11-23T02:16:44.2646857Z 
2022-11-23T02:16:44.2646958Z OK
2022-11-23T02:16:44.2646997Z 
2022-11-23T02:16:44.2647139Z Generating XML reports...
2022-11-23T02:16:44.2647633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015504.xml
2022-11-23T02:16:44.2648042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2648360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2648783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2649014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2649034Z 
2022-11-23T02:16:44.2649179Z Running tests...
2022-11-23T02:16:44.2649479Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2649811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2650126Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2650382Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33251
2022-11-23T02:16:44.2650634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33252
2022-11-23T02:16:44.2651053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2651262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2651680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2651908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2652308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2652560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2652985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2653209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2653493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2653777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2654218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2654652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2654919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2655168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2655543Z STAGE:2022-11-23 01:55:16 33252:33252 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2656350Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:16:44.2656504Z   warnings.warn(
2022-11-23T02:16:44.2656868Z STAGE:2022-11-23 01:55:16 33251:33251 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2657668Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:16:44.2657820Z   warnings.warn(
2022-11-23T02:16:44.2658196Z STAGE:2022-11-23 01:55:16 33251:33251 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2658560Z STAGE:2022-11-23 01:55:16 33252:33252 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2658941Z STAGE:2022-11-23 01:55:16 33251:33251 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2659364Z STAGE:2022-11-23 01:55:16 33252:33252 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2659728Z STAGE:2022-11-23 01:55:16 33251:33251 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2660085Z STAGE:2022-11-23 01:55:16 33252:33252 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2660454Z STAGE:2022-11-23 01:55:16 33251:33251 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2660825Z STAGE:2022-11-23 01:55:16 33252:33252 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2661205Z STAGE:2022-11-23 01:55:16 33251:33251 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2661581Z STAGE:2022-11-23 01:55:16 33252:33252 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2661724Z ok (5.361s)
2022-11-23T02:16:44.2661744Z 
2022-11-23T02:16:44.2662045Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2662175Z Ran 1 test in 5.361s
2022-11-23T02:16:44.2662193Z 
2022-11-23T02:16:44.2662320Z OK
2022-11-23T02:16:44.2662339Z 
2022-11-23T02:16:44.2662497Z Generating XML reports...
2022-11-23T02:16:44.2662986Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015512.xml
2022-11-23T02:16:44.2663393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2663651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2664076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2664302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2664326Z 
2022-11-23T02:16:44.2664471Z Running tests...
2022-11-23T02:16:44.2664756Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2665105Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2665404Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2665659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33366
2022-11-23T02:16:44.2665912Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33367
2022-11-23T02:16:44.2666323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2666535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2666951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2667165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2667568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2667779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2668186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2668409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2668695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2668974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2669413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2669902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2670152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2670417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2670787Z STAGE:2022-11-23 01:55:23 33366:33366 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2671147Z STAGE:2022-11-23 01:55:23 33367:33367 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2671726Z STAGE:2022-11-23 01:55:23 33366:33366 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:55:23 33367:33367 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2671746Z 
2022-11-23T02:16:44.2672347Z STAGE:2022-11-23 01:55:23 33367:33367 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:55:23 33366:33366 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2672370Z 
2022-11-23T02:16:44.2672938Z STAGE:2022-11-23 01:55:23 33366:33366 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:55:23 33367:33367 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2672957Z 
2022-11-23T02:16:44.2673323Z STAGE:2022-11-23 01:55:23 33367:33367 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2673684Z STAGE:2022-11-23 01:55:23 33366:33366 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2674108Z STAGE:2022-11-23 01:55:23 33367:33367 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2674492Z STAGE:2022-11-23 01:55:23 33366:33366 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2674613Z ok (4.261s)
2022-11-23T02:16:44.2674632Z 
2022-11-23T02:16:44.2674936Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2675086Z Ran 1 test in 4.261s
2022-11-23T02:16:44.2675105Z 
2022-11-23T02:16:44.2675233Z OK
2022-11-23T02:16:44.2675252Z 
2022-11-23T02:16:44.2675408Z Generating XML reports...
2022-11-23T02:16:44.2675893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015519.xml
2022-11-23T02:16:44.2676297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2676509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2676913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2677144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2677163Z 
2022-11-23T02:16:44.2677307Z Running tests...
2022-11-23T02:16:44.2677605Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2677959Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2678258Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2678513Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33479
2022-11-23T02:16:44.2678765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33480
2022-11-23T02:16:44.2679172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2679369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2679785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2680011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2680485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2680692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2681102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2681326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2681609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2681873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2682309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2682739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2683010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2683274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2683411Z ok (5.147s)
2022-11-23T02:16:44.2683430Z 
2022-11-23T02:16:44.2683727Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2683874Z Ran 1 test in 5.147s
2022-11-23T02:16:44.2683894Z 
2022-11-23T02:16:44.2684021Z OK
2022-11-23T02:16:44.2684039Z 
2022-11-23T02:16:44.2684179Z Generating XML reports...
2022-11-23T02:16:44.2684712Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015526.xml
2022-11-23T02:16:44.2685125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2685338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2686081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2686321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2686341Z 
2022-11-23T02:16:44.2686487Z Running tests...
2022-11-23T02:16:44.2686791Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2687141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2687412Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2687674Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33590
2022-11-23T02:16:44.2687926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33591
2022-11-23T02:16:44.2688335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2688549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2688962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2689187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2689588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2689779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2690194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2690422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2690703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2690982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2691835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2692590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2693192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2693732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2694394Z STAGE:2022-11-23 01:55:38 33591:33591 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2695028Z STAGE:2022-11-23 01:55:38 33590:33590 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2695684Z STAGE:2022-11-23 01:55:38 33590:33590 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2696344Z STAGE:2022-11-23 01:55:38 33591:33591 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2697016Z STAGE:2022-11-23 01:55:38 33590:33590 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2697671Z STAGE:2022-11-23 01:55:38 33591:33591 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2698324Z STAGE:2022-11-23 01:55:38 33591:33591 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2698964Z STAGE:2022-11-23 01:55:38 33590:33590 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2699879Z STAGE:2022-11-23 01:55:38 33590:33590 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:55:38 33591:33591 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2700315Z 
2022-11-23T02:16:44.2700902Z STAGE:2022-11-23 01:55:38 33590:33590 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:55:38 33591:33591 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2701352Z 
2022-11-23T02:16:44.2701484Z ok (4.213s)
2022-11-23T02:16:44.2701669Z 
2022-11-23T02:16:44.2701995Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2702404Z Ran 1 test in 4.214s
2022-11-23T02:16:44.2702606Z 
2022-11-23T02:16:44.2702715Z OK
2022-11-23T02:16:44.2702888Z 
2022-11-23T02:16:44.2703061Z Generating XML reports...
2022-11-23T02:16:44.2703749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015534.xml
2022-11-23T02:16:44.2704538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2705050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2705716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2706276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2706547Z 
2022-11-23T02:16:44.2706688Z Running tests...
2022-11-23T02:16:44.2707146Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2707749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2708337Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2708886Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33703
2022-11-23T02:16:44.2709411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33704
2022-11-23T02:16:44.2710087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2710609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2711302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2711843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2712490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2713004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2713625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2714163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2714688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2715242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2715986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2716813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2717404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2718050Z STAGE:2022-11-23 01:55:45 33704:33704 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2718616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2719332Z STAGE:2022-11-23 01:55:45 33703:33703 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2720204Z STAGE:2022-11-23 01:55:45 33703:33703 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:55:45 33704:33704 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2720633Z 
2022-11-23T02:16:44.2721242Z STAGE:2022-11-23 01:55:45 33704:33704 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:55:45 33703:33703 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2721686Z 
2022-11-23T02:16:44.2722229Z STAGE:2022-11-23 01:55:45 33704:33704 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:55:45 33703:33703 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2722641Z 
2022-11-23T02:16:44.2723221Z STAGE:2022-11-23 01:55:45 33704:33704 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:55:45 33703:33703 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2723643Z 
2022-11-23T02:16:44.2724027Z STAGE:2022-11-23 01:55:45 33704:33704 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2724706Z STAGE:2022-11-23 01:55:45 33703:33703 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2725131Z ok (4.303s)
2022-11-23T02:16:44.2725295Z 
2022-11-23T02:16:44.2725599Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2726248Z Ran 1 test in 4.304s
2022-11-23T02:16:44.2726448Z 
2022-11-23T02:16:44.2726577Z OK
2022-11-23T02:16:44.2726747Z 
2022-11-23T02:16:44.2726888Z Generating XML reports...
2022-11-23T02:16:44.2727576Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015541.xml
2022-11-23T02:16:44.2728409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2728953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2729601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2730147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2730484Z 
2022-11-23T02:16:44.2730619Z Running tests...
2022-11-23T02:16:44.2731094Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2731695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2732263Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2732827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33816
2022-11-23T02:16:44.2733354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33817
2022-11-23T02:16:44.2734037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2734543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2735190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2735764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2736438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2736951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2737593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2738132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2738724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2739308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2740044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2740817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2741433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2742083Z STAGE:2022-11-23 01:55:51 33817:33817 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2742640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2743293Z STAGE:2022-11-23 01:55:51 33816:33816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2744280Z STAGE:2022-11-23 01:55:51 33816:33816 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:55:51 33817:33817 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2745137Z 
2022-11-23T02:16:44.2746101Z STAGE:2022-11-23 01:55:51 33817:33817 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:55:51 33816:33816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2746561Z 
2022-11-23T02:16:44.2747135Z STAGE:2022-11-23 01:55:51 33816:33816 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:55:51 33817:33817 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2747547Z 
2022-11-23T02:16:44.2747923Z STAGE:2022-11-23 01:55:51 33816:33816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2748598Z STAGE:2022-11-23 01:55:51 33816:33816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2749263Z STAGE:2022-11-23 01:55:51 33817:33817 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2749918Z STAGE:2022-11-23 01:55:51 33817:33817 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2750435Z ok (4.320s)
2022-11-23T02:16:44.2750621Z 
2022-11-23T02:16:44.2750928Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2751309Z Ran 1 test in 4.321s
2022-11-23T02:16:44.2751508Z 
2022-11-23T02:16:44.2751638Z OK
2022-11-23T02:16:44.2751807Z 
2022-11-23T02:16:44.2751967Z Generating XML reports...
2022-11-23T02:16:44.2752647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015548.xml
2022-11-23T02:16:44.2753419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2753947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2754598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2755126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2755395Z 
2022-11-23T02:16:44.2755546Z Running tests...
2022-11-23T02:16:44.2756020Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2756673Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2757237Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2757798Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33929
2022-11-23T02:16:44.2758324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33930
2022-11-23T02:16:44.2759049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2759582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2760233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2760780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2761414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2761935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2762577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2763115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2763626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2764201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2764938Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2765879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2766494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2767044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2767705Z STAGE:2022-11-23 01:55:59 33930:33930 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2768330Z STAGE:2022-11-23 01:55:59 33929:33929 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2768987Z STAGE:2022-11-23 01:55:59 33929:33929 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2769644Z STAGE:2022-11-23 01:55:59 33930:33930 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2770308Z STAGE:2022-11-23 01:55:59 33929:33929 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2771071Z STAGE:2022-11-23 01:55:59 33930:33930 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2771726Z STAGE:2022-11-23 01:55:59 33929:33929 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2772367Z STAGE:2022-11-23 01:55:59 33930:33930 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2773227Z STAGE:2022-11-23 01:56:00 33930:33930 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:56:00 33929:33929 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2773630Z 
2022-11-23T02:16:44.2774242Z STAGE:2022-11-23 01:56:00 33930:33930 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:56:00 33929:33929 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2774685Z 
2022-11-23T02:16:44.2775252Z STAGE:2022-11-23 01:56:00 33930:33930 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:56:00 33929:33929 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2775667Z 
2022-11-23T02:16:44.2776040Z STAGE:2022-11-23 01:56:00 33929:33929 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2776688Z STAGE:2022-11-23 01:56:00 33930:33930 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2777328Z STAGE:2022-11-23 01:56:00 33929:33929 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2778005Z STAGE:2022-11-23 01:56:00 33930:33930 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2778488Z ok (6.230s)
2022-11-23T02:16:44.2778683Z 
2022-11-23T02:16:44.2778988Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2779373Z Ran 1 test in 6.231s
2022-11-23T02:16:44.2779571Z 
2022-11-23T02:16:44.2779700Z OK
2022-11-23T02:16:44.2779870Z 
2022-11-23T02:16:44.2780031Z Generating XML reports...
2022-11-23T02:16:44.2780702Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015554.xml
2022-11-23T02:16:44.2781493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2782015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2782665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2783192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2783459Z 
2022-11-23T02:16:44.2783609Z Running tests...
2022-11-23T02:16:44.2784086Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2784689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2785263Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2785836Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34045
2022-11-23T02:16:44.2786361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34046
2022-11-23T02:16:44.2787021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2787539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2788183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2788727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2789365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2789878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2790584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2791104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2791632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2792203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2792932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2793680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2794281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2794824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2795485Z STAGE:2022-11-23 01:56:08 34046:34046 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2796115Z STAGE:2022-11-23 01:56:08 34045:34045 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2796768Z STAGE:2022-11-23 01:56:08 34045:34045 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2797417Z STAGE:2022-11-23 01:56:08 34046:34046 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2798056Z STAGE:2022-11-23 01:56:08 34045:34045 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2798785Z STAGE:2022-11-23 01:56:08 34046:34046 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2799657Z STAGE:2022-11-23 01:56:08 34045:34045 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:56:08 34046:34046 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2800072Z 
2022-11-23T02:16:44.2800653Z STAGE:2022-11-23 01:56:09 34046:34046 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:56:09 34045:34045 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2801248Z 
2022-11-23T02:16:44.2802521Z STAGE:2022-11-23 01:56:09 34046:34046 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:56:09 34045:34045 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2802977Z 
2022-11-23T02:16:44.2803561Z STAGE:2022-11-23 01:56:09 34046:34046 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:56:09 34045:34045 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2803958Z 
2022-11-23T02:16:44.2804535Z STAGE:2022-11-23 01:56:09 34046:34046 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:56:09 34045:34045 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2804957Z 
2022-11-23T02:16:44.2805563Z STAGE:2022-11-23 01:56:09 34046:34046 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:56:09 34045:34045 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2806240Z 
2022-11-23T02:16:44.2806383Z ok (6.167s)
2022-11-23T02:16:44.2806571Z 
2022-11-23T02:16:44.2806877Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2807251Z Ran 1 test in 6.167s
2022-11-23T02:16:44.2807450Z 
2022-11-23T02:16:44.2807580Z OK
2022-11-23T02:16:44.2807751Z 
2022-11-23T02:16:44.2807915Z Generating XML reports...
2022-11-23T02:16:44.2808601Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015603.xml
2022-11-23T02:16:44.2809371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2810023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2810678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2811207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2811476Z 
2022-11-23T02:16:44.2811623Z Running tests...
2022-11-23T02:16:44.2812096Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2812699Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2813281Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.2813858Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34161
2022-11-23T02:16:44.2814381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34162
2022-11-23T02:16:44.2815046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2815574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2816216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2816806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2817436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2817957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2818664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2819215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2819724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.2820300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.2821032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2821778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.2822377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.2822924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.2823582Z STAGE:2022-11-23 01:56:17 34162:34162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2824212Z STAGE:2022-11-23 01:56:17 34161:34161 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2824866Z STAGE:2022-11-23 01:56:17 34161:34161 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2825514Z STAGE:2022-11-23 01:56:17 34162:34162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2826173Z STAGE:2022-11-23 01:56:17 34161:34161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2826834Z STAGE:2022-11-23 01:56:17 34162:34162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2827489Z STAGE:2022-11-23 01:56:17 34161:34161 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2828129Z STAGE:2022-11-23 01:56:17 34162:34162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2828971Z STAGE:2022-11-23 01:56:18 34162:34162 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:56:18 34161:34161 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2829647Z 
2022-11-23T02:16:44.2830905Z STAGE:2022-11-23 01:56:18 34162:34162 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:56:18 34161:34161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2831355Z 
2022-11-23T02:16:44.2831728Z STAGE:2022-11-23 01:56:18 34162:34162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2832365Z STAGE:2022-11-23 01:56:18 34161:34161 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.2833013Z STAGE:2022-11-23 01:56:18 34162:34162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2833650Z STAGE:2022-11-23 01:56:18 34161:34161 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.2834317Z STAGE:2022-11-23 01:56:18 34162:34162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2834993Z STAGE:2022-11-23 01:56:18 34161:34161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.2835425Z ok (6.142s)
2022-11-23T02:16:44.2835593Z 
2022-11-23T02:16:44.2835895Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2836297Z Ran 1 test in 6.142s
2022-11-23T02:16:44.2836495Z 
2022-11-23T02:16:44.2836626Z OK
2022-11-23T02:16:44.2836778Z 
2022-11-23T02:16:44.2836941Z Generating XML reports...
2022-11-23T02:16:44.2837622Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015612.xml
2022-11-23T02:16:44.2838412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2839004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2839672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2840216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2840488Z 
2022-11-23T02:16:44.2840634Z Running tests...
2022-11-23T02:16:44.2841090Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2841692Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2842256Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.003s)
2022-11-23T02:16:44.2842571Z 
2022-11-23T02:16:44.2842870Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2843245Z Ran 1 test in 0.003s
2022-11-23T02:16:44.2843442Z 
2022-11-23T02:16:44.2843645Z OK (skipped=1)
2022-11-23T02:16:44.2843839Z 
2022-11-23T02:16:44.2843998Z Generating XML reports...
2022-11-23T02:16:44.2844653Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015621.xml
2022-11-23T02:16:44.2845438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2846201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2846856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2847385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2847653Z 
2022-11-23T02:16:44.2847798Z Running tests...
2022-11-23T02:16:44.2848271Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2848855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2849437Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:16:44.2849762Z 
2022-11-23T02:16:44.2850061Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2850455Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2850723Z 
2022-11-23T02:16:44.2850870Z OK (skipped=1)
2022-11-23T02:16:44.2851060Z 
2022-11-23T02:16:44.2851219Z Generating XML reports...
2022-11-23T02:16:44.2851895Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015623.xml
2022-11-23T02:16:44.2852663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2853185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2853832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2854375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2854644Z 
2022-11-23T02:16:44.2854771Z Running tests...
2022-11-23T02:16:44.2855245Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2855840Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2856424Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T02:16:44.2856731Z 
2022-11-23T02:16:44.2857031Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2857428Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2857625Z 
2022-11-23T02:16:44.2857771Z OK (skipped=1)
2022-11-23T02:16:44.2857963Z 
2022-11-23T02:16:44.2858105Z Generating XML reports...
2022-11-23T02:16:44.2858839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015625.xml
2022-11-23T02:16:44.2859638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2860162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2860792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2861339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2861603Z 
2022-11-23T02:16:44.2861749Z Running tests...
2022-11-23T02:16:44.2862202Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2862803Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2863398Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T02:16:44.2863733Z 
2022-11-23T02:16:44.2864035Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2864416Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2864612Z 
2022-11-23T02:16:44.2864752Z OK (skipped=1)
2022-11-23T02:16:44.2864945Z 
2022-11-23T02:16:44.2865100Z Generating XML reports...
2022-11-23T02:16:44.2865747Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015628.xml
2022-11-23T02:16:44.2866542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2867064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2867709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2868231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2868498Z 
2022-11-23T02:16:44.2868644Z Running tests...
2022-11-23T02:16:44.2869117Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2869700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2870282Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:16:44.2870708Z 
2022-11-23T02:16:44.2871005Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2871407Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2871603Z 
2022-11-23T02:16:44.2871728Z OK (skipped=1)
2022-11-23T02:16:44.2871920Z 
2022-11-23T02:16:44.2872080Z Generating XML reports...
2022-11-23T02:16:44.2872750Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015630.xml
2022-11-23T02:16:44.2873544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2874058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2874708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2875247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2875515Z 
2022-11-23T02:16:44.2875646Z Running tests...
2022-11-23T02:16:44.2876117Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2876713Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2877308Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T02:16:44.2877625Z 
2022-11-23T02:16:44.2877924Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2878324Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2878521Z 
2022-11-23T02:16:44.2878666Z OK (skipped=1)
2022-11-23T02:16:44.2878857Z 
2022-11-23T02:16:44.2879047Z Generating XML reports...
2022-11-23T02:16:44.2879731Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015633.xml
2022-11-23T02:16:44.2880519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2881051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2881680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2882220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2882484Z 
2022-11-23T02:16:44.2882629Z Running tests...
2022-11-23T02:16:44.2883083Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2883688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2884269Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:16:44.2884591Z 
2022-11-23T02:16:44.2884883Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2885265Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2885463Z 
2022-11-23T02:16:44.2885607Z OK (skipped=1)
2022-11-23T02:16:44.2886000Z 
2022-11-23T02:16:44.2886156Z Generating XML reports...
2022-11-23T02:16:44.2886843Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015635.xml
2022-11-23T02:16:44.2887615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2888140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2888787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2889317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2889586Z 
2022-11-23T02:16:44.2889732Z Running tests...
2022-11-23T02:16:44.2890211Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2890813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2891481Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:16:44.2891824Z 
2022-11-23T02:16:44.2892125Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2892520Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2892716Z 
2022-11-23T02:16:44.2892843Z OK (skipped=1)
2022-11-23T02:16:44.2893036Z 
2022-11-23T02:16:44.2893195Z Generating XML reports...
2022-11-23T02:16:44.2893870Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015637.xml
2022-11-23T02:16:44.2894662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2895170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2895820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2896372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2896639Z 
2022-11-23T02:16:44.2896783Z Running tests...
2022-11-23T02:16:44.2897241Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2897846Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2898454Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:16:44.2898800Z 
2022-11-23T02:16:44.2899144Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2899550Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2899746Z 
2022-11-23T02:16:44.2899890Z OK (skipped=1)
2022-11-23T02:16:44.2900081Z 
2022-11-23T02:16:44.2900240Z Generating XML reports...
2022-11-23T02:16:44.2900901Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015640.xml
2022-11-23T02:16:44.2901689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2902214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2902842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2903445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2903712Z 
2022-11-23T02:16:44.2903858Z Running tests...
2022-11-23T02:16:44.2904336Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2904922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2905536Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:16:44.2905900Z 
2022-11-23T02:16:44.2906198Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2906578Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2906768Z 
2022-11-23T02:16:44.2906912Z OK (skipped=1)
2022-11-23T02:16:44.2907102Z 
2022-11-23T02:16:44.2907260Z Generating XML reports...
2022-11-23T02:16:44.2907931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015642.xml
2022-11-23T02:16:44.2908698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2909224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2909876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2910421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2910667Z 
2022-11-23T02:16:44.2910870Z Running tests...
2022-11-23T02:16:44.2911349Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2911949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2912546Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:16:44.2912899Z 
2022-11-23T02:16:44.2913199Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2913598Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2913796Z 
2022-11-23T02:16:44.2913940Z OK (skipped=1)
2022-11-23T02:16:44.2914116Z 
2022-11-23T02:16:44.2914273Z Generating XML reports...
2022-11-23T02:16:44.2914946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015645.xml
2022-11-23T02:16:44.2915730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2916238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2916937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2917476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2917741Z 
2022-11-23T02:16:44.2917886Z Running tests...
2022-11-23T02:16:44.2918347Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2918944Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2919619Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:16:44.2920231Z 
2022-11-23T02:16:44.2920862Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2921476Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2921681Z 
2022-11-23T02:16:44.2921831Z OK (skipped=1)
2022-11-23T02:16:44.2922023Z 
2022-11-23T02:16:44.2922184Z Generating XML reports...
2022-11-23T02:16:44.2922851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015647.xml
2022-11-23T02:16:44.2923645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2924169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2924822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2925349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2925616Z 
2022-11-23T02:16:44.2926003Z Running tests...
2022-11-23T02:16:44.2926494Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2927077Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2927706Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:16:44.2928066Z 
2022-11-23T02:16:44.2928364Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2928764Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2928965Z 
2022-11-23T02:16:44.2929090Z OK (skipped=1)
2022-11-23T02:16:44.2929282Z 
2022-11-23T02:16:44.2929442Z Generating XML reports...
2022-11-23T02:16:44.2930120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015649.xml
2022-11-23T02:16:44.2930908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2931410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2932063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2932712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2932980Z 
2022-11-23T02:16:44.2933106Z Running tests...
2022-11-23T02:16:44.2933588Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2934191Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2934824Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:16:44.2935197Z 
2022-11-23T02:16:44.2935483Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2935881Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2936078Z 
2022-11-23T02:16:44.2936226Z OK (skipped=1)
2022-11-23T02:16:44.2936417Z 
2022-11-23T02:16:44.2936577Z Generating XML reports...
2022-11-23T02:16:44.2937235Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015652.xml
2022-11-23T02:16:44.2938022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2938544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2939170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2939721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2939989Z 
2022-11-23T02:16:44.2940200Z Running tests...
2022-11-23T02:16:44.2940687Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2941271Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2941879Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:16:44.2942243Z 
2022-11-23T02:16:44.2942540Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2942939Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2943117Z 
2022-11-23T02:16:44.2943254Z OK (skipped=1)
2022-11-23T02:16:44.2943443Z 
2022-11-23T02:16:44.2943601Z Generating XML reports...
2022-11-23T02:16:44.2944281Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015654.xml
2022-11-23T02:16:44.2945050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2945575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2946227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2946771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2947026Z 
2022-11-23T02:16:44.2947175Z Running tests...
2022-11-23T02:16:44.2947651Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2948250Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2948855Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:16:44.2949218Z 
2022-11-23T02:16:44.2949520Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2949923Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2950121Z 
2022-11-23T02:16:44.2950270Z OK (skipped=1)
2022-11-23T02:16:44.2950462Z 
2022-11-23T02:16:44.2950603Z Generating XML reports...
2022-11-23T02:16:44.2951277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015657.xml
2022-11-23T02:16:44.2952147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2952673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2953304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2953856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2954123Z 
2022-11-23T02:16:44.2954267Z Running tests...
2022-11-23T02:16:44.2954721Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2955332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2955945Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:16:44.2956299Z 
2022-11-23T02:16:44.2956596Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2956986Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2957183Z 
2022-11-23T02:16:44.2957327Z OK (skipped=1)
2022-11-23T02:16:44.2957517Z 
2022-11-23T02:16:44.2957677Z Generating XML reports...
2022-11-23T02:16:44.2958335Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015659.xml
2022-11-23T02:16:44.2959124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2959650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2960349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2960889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2961157Z 
2022-11-23T02:16:44.2961302Z Running tests...
2022-11-23T02:16:44.2961811Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2962420Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2963025Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:16:44.2963387Z 
2022-11-23T02:16:44.2963686Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2964080Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2964278Z 
2022-11-23T02:16:44.2964403Z OK (skipped=1)
2022-11-23T02:16:44.2964595Z 
2022-11-23T02:16:44.2964758Z Generating XML reports...
2022-11-23T02:16:44.2965437Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015701.xml
2022-11-23T02:16:44.2966489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2966988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2967643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2968185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2968453Z 
2022-11-23T02:16:44.2968599Z Running tests...
2022-11-23T02:16:44.2969056Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2969657Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2970278Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:16:44.2970643Z 
2022-11-23T02:16:44.2970921Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2971312Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2971503Z 
2022-11-23T02:16:44.2971645Z OK (skipped=1)
2022-11-23T02:16:44.2971968Z 
2022-11-23T02:16:44.2972125Z Generating XML reports...
2022-11-23T02:16:44.2972786Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015704.xml
2022-11-23T02:16:44.2973574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2974100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2974728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2975278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2975546Z 
2022-11-23T02:16:44.2975692Z Running tests...
2022-11-23T02:16:44.2976166Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2976748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2977383Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:16:44.2977754Z 
2022-11-23T02:16:44.2978054Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2978452Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2978628Z 
2022-11-23T02:16:44.2978766Z OK (skipped=1)
2022-11-23T02:16:44.2978957Z 
2022-11-23T02:16:44.2979111Z Generating XML reports...
2022-11-23T02:16:44.2979785Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015706.xml
2022-11-23T02:16:44.2980616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2981148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2981801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2982348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2982615Z 
2022-11-23T02:16:44.2982741Z Running tests...
2022-11-23T02:16:44.2983215Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2983814Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2984448Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:16:44.2984792Z 
2022-11-23T02:16:44.2985097Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2985495Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2985693Z 
2022-11-23T02:16:44.2985838Z OK (skipped=1)
2022-11-23T02:16:44.2985857Z 
2022-11-23T02:16:44.2986017Z Generating XML reports...
2022-11-23T02:16:44.2986485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015709.xml
2022-11-23T02:16:44.2986900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2987115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2987533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2987763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2987782Z 
2022-11-23T02:16:44.2987917Z Running tests...
2022-11-23T02:16:44.2988216Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2988569Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2988915Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:16:44.2988986Z 
2022-11-23T02:16:44.2989298Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2989427Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2989447Z 
2022-11-23T02:16:44.2989590Z OK (skipped=1)
2022-11-23T02:16:44.2989609Z 
2022-11-23T02:16:44.2989768Z Generating XML reports...
2022-11-23T02:16:44.2990251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015711.xml
2022-11-23T02:16:44.2990662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2990874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2991290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2991518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2991541Z 
2022-11-23T02:16:44.2991662Z Running tests...
2022-11-23T02:16:44.2991960Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2992311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2992643Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:16:44.2992663Z 
2022-11-23T02:16:44.2992955Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2993100Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2993119Z 
2022-11-23T02:16:44.2993256Z OK (skipped=1)
2022-11-23T02:16:44.2993320Z 
2022-11-23T02:16:44.2993483Z Generating XML reports...
2022-11-23T02:16:44.2993973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015713.xml
2022-11-23T02:16:44.2994362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2994577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2994997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2995224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2995244Z 
2022-11-23T02:16:44.2995391Z Running tests...
2022-11-23T02:16:44.2995689Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2996042Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.2996384Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:16:44.2996403Z 
2022-11-23T02:16:44.2996696Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2996830Z Ran 1 test in 0.002s
2022-11-23T02:16:44.2996868Z 
2022-11-23T02:16:44.2996983Z OK (skipped=1)
2022-11-23T02:16:44.2997003Z 
2022-11-23T02:16:44.2997160Z Generating XML reports...
2022-11-23T02:16:44.2997639Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015716.xml
2022-11-23T02:16:44.2998047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.2998259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.2998672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.2998902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.2998921Z 
2022-11-23T02:16:44.2999065Z Running tests...
2022-11-23T02:16:44.2999344Z ----------------------------------------------------------------------
2022-11-23T02:16:44.2999749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3000047Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3000312Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35069
2022-11-23T02:16:44.3000564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35070
2022-11-23T02:16:44.3000971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3001180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3001599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3001806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3002209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3002423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3002831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3003056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3003339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3003622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3004104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3004548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3004795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3005063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3005338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3005618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3006285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3006723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3006859Z ok (5.629s)
2022-11-23T02:16:44.3006878Z 
2022-11-23T02:16:44.3007182Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3007331Z Ran 1 test in 5.629s
2022-11-23T02:16:44.3007350Z 
2022-11-23T02:16:44.3007465Z OK
2022-11-23T02:16:44.3007507Z 
2022-11-23T02:16:44.3007650Z Generating XML reports...
2022-11-23T02:16:44.3008138Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015718.xml
2022-11-23T02:16:44.3008550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3008764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3009183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3009412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3009432Z 
2022-11-23T02:16:44.3009568Z Running tests...
2022-11-23T02:16:44.3009869Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3010199Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3010579Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3010837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35190
2022-11-23T02:16:44.3011094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35191
2022-11-23T02:16:44.3011506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3011718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3012140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3012367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3012745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3012961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3013372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3013598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3013880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3014160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3014654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3015101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3015366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3015614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3015803Z skip: Need at least 3 CUDA devices (4.166s)
2022-11-23T02:16:44.3015822Z 
2022-11-23T02:16:44.3016129Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3016278Z Ran 1 test in 4.166s
2022-11-23T02:16:44.3016298Z 
2022-11-23T02:16:44.3016491Z OK (skipped=1)
2022-11-23T02:16:44.3016510Z 
2022-11-23T02:16:44.3016672Z Generating XML reports...
2022-11-23T02:16:44.3017158Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015726.xml
2022-11-23T02:16:44.3017569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3017783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3018179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3018415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3018434Z 
2022-11-23T02:16:44.3018581Z Running tests...
2022-11-23T02:16:44.3018883Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3019234Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3019519Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-11-23T02:16:44.3019539Z 
2022-11-23T02:16:44.3019836Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3019986Z Ran 1 test in 0.002s
2022-11-23T02:16:44.3020004Z 
2022-11-23T02:16:44.3020139Z OK (skipped=1)
2022-11-23T02:16:44.3020157Z 
2022-11-23T02:16:44.3020298Z Generating XML reports...
2022-11-23T02:16:44.3020782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015733.xml
2022-11-23T02:16:44.3021251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3021460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3021878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3022103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3022123Z 
2022-11-23T02:16:44.3022261Z Running tests...
2022-11-23T02:16:44.3022564Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3022917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3023176Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3023435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35332
2022-11-23T02:16:44.3023694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35333
2022-11-23T02:16:44.3024104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3024314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3024729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3024954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3025401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3025594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3026002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3026234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3026518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3026800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3027239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3027669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3027941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3028204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3028314Z ok (5.173s)
2022-11-23T02:16:44.3028353Z 
2022-11-23T02:16:44.3028636Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3028783Z Ran 1 test in 5.174s
2022-11-23T02:16:44.3028802Z 
2022-11-23T02:16:44.3028930Z OK
2022-11-23T02:16:44.3028949Z 
2022-11-23T02:16:44.3029110Z Generating XML reports...
2022-11-23T02:16:44.3029594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015735.xml
2022-11-23T02:16:44.3030005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3030215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3030639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3030849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3030869Z 
2022-11-23T02:16:44.3031018Z Running tests...
2022-11-23T02:16:44.3031315Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3031720Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3032008Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3032264Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35441
2022-11-23T02:16:44.3032518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35442
2022-11-23T02:16:44.3032927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3033139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3033535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3033759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3034159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3034369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3034781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3035000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3035285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3035606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3036035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3036469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3036741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3037001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3037143Z ok (5.926s)
2022-11-23T02:16:44.3037163Z 
2022-11-23T02:16:44.3037461Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3037600Z Ran 1 test in 5.927s
2022-11-23T02:16:44.3037619Z 
2022-11-23T02:16:44.3037746Z OK
2022-11-23T02:16:44.3037764Z 
2022-11-23T02:16:44.3037917Z Generating XML reports...
2022-11-23T02:16:44.3038387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015743.xml
2022-11-23T02:16:44.3038798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3039015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3039431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3039659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3039678Z 
2022-11-23T02:16:44.3039816Z Running tests...
2022-11-23T02:16:44.3040115Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3040463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3040761Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3041000Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35552
2022-11-23T02:16:44.3041252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35553
2022-11-23T02:16:44.3041661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3041948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3042367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3042590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3042986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3043191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3043582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3043811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3044087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3044363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3044803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3045235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3045501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3045984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3046346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3046616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3047057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3047499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3047635Z ok (4.964s)
2022-11-23T02:16:44.3047655Z 
2022-11-23T02:16:44.3047953Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3048091Z Ran 1 test in 4.964s
2022-11-23T02:16:44.3048111Z 
2022-11-23T02:16:44.3048237Z OK
2022-11-23T02:16:44.3048255Z 
2022-11-23T02:16:44.3048413Z Generating XML reports...
2022-11-23T02:16:44.3048902Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015752.xml
2022-11-23T02:16:44.3049294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3049510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3049926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3050160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3050180Z 
2022-11-23T02:16:44.3050323Z Running tests...
2022-11-23T02:16:44.3050617Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3050969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3051273Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3051530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35667
2022-11-23T02:16:44.3051768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35668
2022-11-23T02:16:44.3052179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3052386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3052868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3053098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3053498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3053703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3054120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3054345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3054609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3054886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3055325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3055762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3056024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3056293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3056487Z skip: Skipped due to small world size. (4.346s)
2022-11-23T02:16:44.3056507Z 
2022-11-23T02:16:44.3056851Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3056976Z Ran 1 test in 4.346s
2022-11-23T02:16:44.3057019Z 
2022-11-23T02:16:44.3057144Z OK (skipped=1)
2022-11-23T02:16:44.3057163Z 
2022-11-23T02:16:44.3057319Z Generating XML reports...
2022-11-23T02:16:44.3057806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015759.xml
2022-11-23T02:16:44.3058214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3058420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3058835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3059059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3059078Z 
2022-11-23T02:16:44.3059221Z Running tests...
2022-11-23T02:16:44.3059506Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3059858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3060151Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3060402Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35776
2022-11-23T02:16:44.3060661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35777
2022-11-23T02:16:44.3061065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3061277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3061688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3061912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3062296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3062511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3062924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3063203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3063482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3063765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3064205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3064637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3064904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3065149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3065347Z skip: Skipped due to small world size. (4.210s)
2022-11-23T02:16:44.3065366Z 
2022-11-23T02:16:44.3065670Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3065814Z Ran 1 test in 4.210s
2022-11-23T02:16:44.3065833Z 
2022-11-23T02:16:44.3065981Z OK (skipped=1)
2022-11-23T02:16:44.3066000Z 
2022-11-23T02:16:44.3066150Z Generating XML reports...
2022-11-23T02:16:44.3066632Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015806.xml
2022-11-23T02:16:44.3067038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3067277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3067696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3067925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3067944Z 
2022-11-23T02:16:44.3068087Z Running tests...
2022-11-23T02:16:44.3068394Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3068739Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3069039Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3069294Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35885
2022-11-23T02:16:44.3069555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35886
2022-11-23T02:16:44.3069949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3070159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3070575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3070800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3071207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3071413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3071826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3072046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3072308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3072595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3073034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3073462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3073786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3074048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3074245Z skip: Skipped due to small world size. (4.221s)
2022-11-23T02:16:44.3074265Z 
2022-11-23T02:16:44.3074565Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3074711Z Ran 1 test in 4.221s
2022-11-23T02:16:44.3074730Z 
2022-11-23T02:16:44.3074847Z OK (skipped=1)
2022-11-23T02:16:44.3074884Z 
2022-11-23T02:16:44.3075028Z Generating XML reports...
2022-11-23T02:16:44.3075513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015813.xml
2022-11-23T02:16:44.3075920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3076136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3076549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3076772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3076792Z 
2022-11-23T02:16:44.3076939Z Running tests...
2022-11-23T02:16:44.3077234Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3077561Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3077912Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3078175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35994
2022-11-23T02:16:44.3078424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35995
2022-11-23T02:16:44.3078837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3079045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3079454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3079680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3080057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3080271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3080688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3080914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3081192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3081476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3081910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3082341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3082606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3082856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3083132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3083409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3083840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3084321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3084461Z ok (5.268s)
2022-11-23T02:16:44.3084480Z 
2022-11-23T02:16:44.3084780Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3084923Z Ran 1 test in 5.268s
2022-11-23T02:16:44.3084942Z 
2022-11-23T02:16:44.3085072Z OK
2022-11-23T02:16:44.3085090Z 
2022-11-23T02:16:44.3085232Z Generating XML reports...
2022-11-23T02:16:44.3085956Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015819.xml
2022-11-23T02:16:44.3086381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3086593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3087010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3087235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3087255Z 
2022-11-23T02:16:44.3087401Z Running tests...
2022-11-23T02:16:44.3087700Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3088047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3088331Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3088701Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36109
2022-11-23T02:16:44.3088967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36110
2022-11-23T02:16:44.3089378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3089592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3090010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3090237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3090643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3090832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3091249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3091474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3091756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3092035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3092473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3092902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3093168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3093431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3093692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3093971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3094406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3094999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3095225Z ok (5.308s)
2022-11-23T02:16:44.3095259Z 
2022-11-23T02:16:44.3095890Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3096161Z Ran 1 test in 5.308s
2022-11-23T02:16:44.3096192Z 
2022-11-23T02:16:44.3096402Z OK
2022-11-23T02:16:44.3096429Z 
2022-11-23T02:16:44.3096607Z Generating XML reports...
2022-11-23T02:16:44.3097089Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015827.xml
2022-11-23T02:16:44.3097495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3097705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3098119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3098349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3098368Z 
2022-11-23T02:16:44.3098511Z Running tests...
2022-11-23T02:16:44.3098811Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3099158Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3099440Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3099755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36224
2022-11-23T02:16:44.3100016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36225
2022-11-23T02:16:44.3100424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3100632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3101050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3101275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3101673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3101880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3102277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3102505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3102785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3103060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3103500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3103928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3104191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3104453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3104630Z skip: Skipped due to small world size. (4.241s)
2022-11-23T02:16:44.3104666Z 
2022-11-23T02:16:44.3104954Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3105103Z Ran 1 test in 4.241s
2022-11-23T02:16:44.3105122Z 
2022-11-23T02:16:44.3105265Z OK (skipped=1)
2022-11-23T02:16:44.3105284Z 
2022-11-23T02:16:44.3105443Z Generating XML reports...
2022-11-23T02:16:44.3105930Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015835.xml
2022-11-23T02:16:44.3106387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3106598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3107011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3107220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3107255Z 
2022-11-23T02:16:44.3107384Z Running tests...
2022-11-23T02:16:44.3107686Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3108033Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3108398Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3108655Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36333
2022-11-23T02:16:44.3108908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36334
2022-11-23T02:16:44.3109313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3109522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3109918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3110188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3110590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3110799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3111203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3111429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3111709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3111985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3112405Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3112840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3113103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3113362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3113498Z ok (4.355s)
2022-11-23T02:16:44.3113522Z 
2022-11-23T02:16:44.3113820Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3113968Z Ran 1 test in 4.355s
2022-11-23T02:16:44.3113986Z 
2022-11-23T02:16:44.3114113Z OK
2022-11-23T02:16:44.3114132Z 
2022-11-23T02:16:44.3114290Z Generating XML reports...
2022-11-23T02:16:44.3114761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015842.xml
2022-11-23T02:16:44.3115164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3115378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3115791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3116017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3116038Z 
2022-11-23T02:16:44.3116174Z Running tests...
2022-11-23T02:16:44.3116591Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3116946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3117237Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3117491Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36442
2022-11-23T02:16:44.3117742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36443
2022-11-23T02:16:44.3118151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3118360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3118769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3118991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3119395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3119603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3119997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3120220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3120498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3120838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3121278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3121706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3121974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3122235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3122372Z ok (4.254s)
2022-11-23T02:16:44.3122392Z 
2022-11-23T02:16:44.3122677Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3122825Z Ran 1 test in 4.255s
2022-11-23T02:16:44.3122844Z 
2022-11-23T02:16:44.3122972Z OK
2022-11-23T02:16:44.3122990Z 
2022-11-23T02:16:44.3123151Z Generating XML reports...
2022-11-23T02:16:44.3123639Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015849.xml
2022-11-23T02:16:44.3124039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3124254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3124665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3124874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3124910Z 
2022-11-23T02:16:44.3125036Z Running tests...
2022-11-23T02:16:44.3125330Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3125906Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3126231Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:16:44.3126251Z 
2022-11-23T02:16:44.3126555Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3126698Z Ran 1 test in 0.002s
2022-11-23T02:16:44.3126718Z 
2022-11-23T02:16:44.3126851Z OK (skipped=1)
2022-11-23T02:16:44.3126947Z 
2022-11-23T02:16:44.3127112Z Generating XML reports...
2022-11-23T02:16:44.3127583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015856.xml
2022-11-23T02:16:44.3127988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3128196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3128603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3128828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3128847Z 
2022-11-23T02:16:44.3128983Z Running tests...
2022-11-23T02:16:44.3129281Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3129627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3129905Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-11-23T02:16:44.3129942Z 
2022-11-23T02:16:44.3130215Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3130362Z Ran 1 test in 0.003s
2022-11-23T02:16:44.3130381Z 
2022-11-23T02:16:44.3130513Z OK (skipped=1)
2022-11-23T02:16:44.3130531Z 
2022-11-23T02:16:44.3130689Z Generating XML reports...
2022-11-23T02:16:44.3131170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015858.xml
2022-11-23T02:16:44.3131637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3131855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3132269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3132481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3132518Z 
2022-11-23T02:16:44.3132645Z Running tests...
2022-11-23T02:16:44.3132942Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3133292Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3133596Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-11-23T02:16:44.3133615Z 
2022-11-23T02:16:44.3133908Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3134057Z Ran 1 test in 0.003s
2022-11-23T02:16:44.3134076Z 
2022-11-23T02:16:44.3134218Z OK (skipped=1)
2022-11-23T02:16:44.3134237Z 
2022-11-23T02:16:44.3134392Z Generating XML reports...
2022-11-23T02:16:44.3134858Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015900.xml
2022-11-23T02:16:44.3135268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3135478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3135889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3136113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3136132Z 
2022-11-23T02:16:44.3136275Z Running tests...
2022-11-23T02:16:44.3136572Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3136919Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3137212Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:16:44.3137231Z 
2022-11-23T02:16:44.3137502Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3137701Z Ran 1 test in 0.002s
2022-11-23T02:16:44.3137721Z 
2022-11-23T02:16:44.3137863Z OK (skipped=1)
2022-11-23T02:16:44.3137881Z 
2022-11-23T02:16:44.3138040Z Generating XML reports...
2022-11-23T02:16:44.3138524Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015903.xml
2022-11-23T02:16:44.3138927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3139135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3139554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3139779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3139798Z 
2022-11-23T02:16:44.3139925Z Running tests...
2022-11-23T02:16:44.3140220Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3140572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3140870Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:16:44.3140889Z 
2022-11-23T02:16:44.3141179Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3141326Z Ran 1 test in 0.002s
2022-11-23T02:16:44.3141344Z 
2022-11-23T02:16:44.3141484Z OK (skipped=1)
2022-11-23T02:16:44.3141503Z 
2022-11-23T02:16:44.3141660Z Generating XML reports...
2022-11-23T02:16:44.3142167Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015905.xml
2022-11-23T02:16:44.3142578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3142797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3143213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3143437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3143456Z 
2022-11-23T02:16:44.3143599Z Running tests...
2022-11-23T02:16:44.3143892Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3144239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3144552Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:16:44.3144572Z 
2022-11-23T02:16:44.3144849Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3144996Z Ran 1 test in 0.003s
2022-11-23T02:16:44.3145014Z 
2022-11-23T02:16:44.3145147Z OK (skipped=1)
2022-11-23T02:16:44.3145165Z 
2022-11-23T02:16:44.3145327Z Generating XML reports...
2022-11-23T02:16:44.3145812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015908.xml
2022-11-23T02:16:44.3146211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3146419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3146833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3147063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3147083Z 
2022-11-23T02:16:44.3147211Z Running tests...
2022-11-23T02:16:44.3147508Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3147855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3148157Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:16:44.3148225Z 
2022-11-23T02:16:44.3148524Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3148671Z Ran 1 test in 0.003s
2022-11-23T02:16:44.3148691Z 
2022-11-23T02:16:44.3148829Z OK (skipped=1)
2022-11-23T02:16:44.3148847Z 
2022-11-23T02:16:44.3149003Z Generating XML reports...
2022-11-23T02:16:44.3149466Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015910.xml
2022-11-23T02:16:44.3149874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3150083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3150491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3150719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3150737Z 
2022-11-23T02:16:44.3150880Z Running tests...
2022-11-23T02:16:44.3151170Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3151516Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3151815Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:16:44.3151835Z 
2022-11-23T02:16:44.3152112Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3152259Z Ran 1 test in 0.002s
2022-11-23T02:16:44.3152324Z 
2022-11-23T02:16:44.3152475Z OK (skipped=1)
2022-11-23T02:16:44.3152494Z 
2022-11-23T02:16:44.3152653Z Generating XML reports...
2022-11-23T02:16:44.3153135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015912.xml
2022-11-23T02:16:44.3153538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3153746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3154157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3154381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3154400Z 
2022-11-23T02:16:44.3154526Z Running tests...
2022-11-23T02:16:44.3154824Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3155174Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3155456Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3155709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36815
2022-11-23T02:16:44.3155968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36816
2022-11-23T02:16:44.3156369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3156577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3156972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3157196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3157592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3157799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3158207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3158433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3158767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3159047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3159485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3159898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3160168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3160433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3160803Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3161156Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3161745Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3161765Z 
2022-11-23T02:16:44.3162365Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3162385Z 
2022-11-23T02:16:44.3162789Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3163152Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3163520Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3164105Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3164129Z 
2022-11-23T02:16:44.3164495Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3164850Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3165203Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3165574Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3166268Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3166658Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3167047Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3167406Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3167763Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3168114Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3168494Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3168862Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3169244Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3169606Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3170054Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3170419Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3170779Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3171154Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3171514Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3171874Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3172226Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3172591Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3173184Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3173205Z 
2022-11-23T02:16:44.3173577Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3173927Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3174336Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3174709Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3175065Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3175426Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3175806Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3176164Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3176514Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3176876Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3177252Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3177611Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3177985Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3178349Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3178687Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3179052Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3179410Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3179784Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3180161Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3180519Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3180873Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3181291Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3181876Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3181896Z 
2022-11-23T02:16:44.3182271Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3182609Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3182964Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3183327Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3183702Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3184068Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3184444Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3184801Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3185154Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3185563Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3185931Z STAGE:2022-11-23 01:59:19 36816:36816 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3186295Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3186669Z STAGE:2022-11-23 01:59:19 36815:36815 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3186812Z ok (4.253s)
2022-11-23T02:16:44.3186831Z 
2022-11-23T02:16:44.3187129Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3187276Z Ran 1 test in 4.254s
2022-11-23T02:16:44.3187295Z 
2022-11-23T02:16:44.3187421Z OK
2022-11-23T02:16:44.3187439Z 
2022-11-23T02:16:44.3187597Z Generating XML reports...
2022-11-23T02:16:44.3188072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015915.xml
2022-11-23T02:16:44.3188482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3188695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3189109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3189338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3189357Z 
2022-11-23T02:16:44.3189500Z Running tests...
2022-11-23T02:16:44.3189796Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3190145Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3190436Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3191217Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.601s)
2022-11-23T02:16:44.3191258Z 
2022-11-23T02:16:44.3191536Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3191684Z Ran 1 test in 1.601s
2022-11-23T02:16:44.3191763Z 
2022-11-23T02:16:44.3191910Z OK (skipped=1)
2022-11-23T02:16:44.3191928Z 
2022-11-23T02:16:44.3192086Z Generating XML reports...
2022-11-23T02:16:44.3192572Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015922.xml
2022-11-23T02:16:44.3192979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3193188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3193607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3193834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3193853Z 
2022-11-23T02:16:44.3193980Z Running tests...
2022-11-23T02:16:44.3194280Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3194629Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3194930Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3195184Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36962
2022-11-23T02:16:44.3195437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36963
2022-11-23T02:16:44.3195843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3196052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3196493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3196727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3197132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3197342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3197746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3197969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3198248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3198524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3198962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3199378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3199646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3199912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3200186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3200461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3200895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3201324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3201695Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3202053Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3202406Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3202828Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3203214Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3203648Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3204010Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3204366Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3204732Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3205100Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3205481Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3206058Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3206430Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3206786Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3207152Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3207586Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3207974Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3208354Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3208716Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3209076Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3209423Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3209786Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3210163Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3210542Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3210899Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3211251Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3211623Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3211996Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3212363Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3212720Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3213080Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3213436Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3213804Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3214162Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3214614Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3214990Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3215349Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3215702Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3216052Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3216679Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3216700Z 
2022-11-23T02:16:44.3217080Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3217437Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3217788Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3218152Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3218512Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3218936Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3219321Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3219660Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3220023Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3220386Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3220760Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3221121Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3221494Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3221854Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3222205Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3222567Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3222913Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3223600Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3224417Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3224899Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3225263Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3225631Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3226007Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3226370Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3226815Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3227155Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3227507Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3227873Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3228242Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3228617Z STAGE:2022-11-23 01:59:29 36963:36963 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3228992Z STAGE:2022-11-23 01:59:29 36962:36962 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3229132Z ok (4.232s)
2022-11-23T02:16:44.3229156Z 
2022-11-23T02:16:44.3229456Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3229604Z Ran 1 test in 4.232s
2022-11-23T02:16:44.3229623Z 
2022-11-23T02:16:44.3229733Z OK
2022-11-23T02:16:44.3229751Z 
2022-11-23T02:16:44.3229910Z Generating XML reports...
2022-11-23T02:16:44.3230397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015926.xml
2022-11-23T02:16:44.3230802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3231057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3231479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3231704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3231724Z 
2022-11-23T02:16:44.3231868Z Running tests...
2022-11-23T02:16:44.3232166Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3232497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3232791Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3233045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37081
2022-11-23T02:16:44.3233296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37082
2022-11-23T02:16:44.3233706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3233914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3234330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3234557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3234943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3235152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3235562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3235784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3236064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3236343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3236781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3237214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3237536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3237780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3237973Z skip: Skipped due to small world size. (4.151s)
2022-11-23T02:16:44.3237992Z 
2022-11-23T02:16:44.3238296Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3238443Z Ran 1 test in 4.151s
2022-11-23T02:16:44.3238462Z 
2022-11-23T02:16:44.3238605Z OK (skipped=1)
2022-11-23T02:16:44.3238624Z 
2022-11-23T02:16:44.3238786Z Generating XML reports...
2022-11-23T02:16:44.3239272Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015932.xml
2022-11-23T02:16:44.3239679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3239877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3240292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3240518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3240537Z 
2022-11-23T02:16:44.3240669Z Running tests...
2022-11-23T02:16:44.3240964Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3241314Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3241655Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3241914Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37190
2022-11-23T02:16:44.3242166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37191
2022-11-23T02:16:44.3242563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3242773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3243188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3243414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3243815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3244025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3244438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3244660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3244924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3245210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3245648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3246333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3246599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3246866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3247681Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:16:44.3247918Z   warnings.warn(
2022-11-23T02:16:44.3248725Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:16:44.3248870Z   warnings.warn(
2022-11-23T02:16:44.3248990Z ok (5.037s)
2022-11-23T02:16:44.3249032Z 
2022-11-23T02:16:44.3249314Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3249462Z Ran 1 test in 5.037s
2022-11-23T02:16:44.3249485Z 
2022-11-23T02:16:44.3249616Z OK
2022-11-23T02:16:44.3249634Z 
2022-11-23T02:16:44.3249793Z Generating XML reports...
2022-11-23T02:16:44.3250282Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015939.xml
2022-11-23T02:16:44.3250686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3250898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3251313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3251522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3251541Z 
2022-11-23T02:16:44.3251685Z Running tests...
2022-11-23T02:16:44.3251983Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3252392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3252701Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3253491Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.631s)
2022-11-23T02:16:44.3253516Z 
2022-11-23T02:16:44.3253810Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3253957Z Ran 1 test in 1.631s
2022-11-23T02:16:44.3253976Z 
2022-11-23T02:16:44.3254119Z OK (skipped=1)
2022-11-23T02:16:44.3254139Z 
2022-11-23T02:16:44.3254281Z Generating XML reports...
2022-11-23T02:16:44.3254765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015947.xml
2022-11-23T02:16:44.3255172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3255382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3255794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3256023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3256042Z 
2022-11-23T02:16:44.3256186Z Running tests...
2022-11-23T02:16:44.3256481Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3256830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3257164Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3257955Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.624s)
2022-11-23T02:16:44.3257975Z 
2022-11-23T02:16:44.3258273Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3258505Z Ran 1 test in 1.624s
2022-11-23T02:16:44.3258524Z 
2022-11-23T02:16:44.3258673Z OK (skipped=1)
2022-11-23T02:16:44.3258691Z 
2022-11-23T02:16:44.3258850Z Generating XML reports...
2022-11-23T02:16:44.3259333Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015951.xml
2022-11-23T02:16:44.3259733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3259941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3260359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3260568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3260589Z 
2022-11-23T02:16:44.3260732Z Running tests...
2022-11-23T02:16:44.3261027Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3261378Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3261728Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3262505Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.639s)
2022-11-23T02:16:44.3262570Z 
2022-11-23T02:16:44.3262870Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3263015Z Ran 1 test in 1.639s
2022-11-23T02:16:44.3263034Z 
2022-11-23T02:16:44.3263175Z OK (skipped=1)
2022-11-23T02:16:44.3263194Z 
2022-11-23T02:16:44.3263351Z Generating XML reports...
2022-11-23T02:16:44.3263826Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015955.xml
2022-11-23T02:16:44.3264228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3264438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3264853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3265150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3265182Z 
2022-11-23T02:16:44.3265423Z Running tests...
2022-11-23T02:16:44.3265820Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3266171Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3266454Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3266713Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37403
2022-11-23T02:16:44.3266965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37404
2022-11-23T02:16:44.3267371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3267579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3267988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3268217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3268618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3268824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3269300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3269524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3269802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3270076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3270510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3270943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3271206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3271466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3271760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05o0zint
2022-11-23T02:16:44.3272048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05o0zint/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3272334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpske_ayik
2022-11-23T02:16:44.3272632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpske_ayik/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3272900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3273215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3273359Z ok (5.548s)
2022-11-23T02:16:44.3273378Z 
2022-11-23T02:16:44.3273682Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3273827Z Ran 1 test in 5.548s
2022-11-23T02:16:44.3273846Z 
2022-11-23T02:16:44.3273960Z OK
2022-11-23T02:16:44.3273978Z 
2022-11-23T02:16:44.3274137Z Generating XML reports...
2022-11-23T02:16:44.3274621Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015959.xml
2022-11-23T02:16:44.3275025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3275237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3275650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3275876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3275896Z 
2022-11-23T02:16:44.3276039Z Running tests...
2022-11-23T02:16:44.3276333Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3276664Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3276977Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3277228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37518
2022-11-23T02:16:44.3277477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37519
2022-11-23T02:16:44.3277878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3278086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3278502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3278728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3279113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3279378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3279788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3280009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3280288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3280563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3281001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3281433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3281695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3281942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3282232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpokcg28b8
2022-11-23T02:16:44.3282537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpokcg28b8/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3282823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppivp6ph1
2022-11-23T02:16:44.3283124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppivp6ph1/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3283436Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3283714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3283974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3284225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3284367Z ok (5.631s)
2022-11-23T02:16:44.3284388Z 
2022-11-23T02:16:44.3284693Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3284839Z Ran 1 test in 5.631s
2022-11-23T02:16:44.3284858Z 
2022-11-23T02:16:44.3284983Z OK
2022-11-23T02:16:44.3285001Z 
2022-11-23T02:16:44.3285157Z Generating XML reports...
2022-11-23T02:16:44.3285643Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020007.xml
2022-11-23T02:16:44.3286305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3286517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3286915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3287140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3287163Z 
2022-11-23T02:16:44.3287305Z Running tests...
2022-11-23T02:16:44.3287601Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3287947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3288253Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3289040Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s)
2022-11-23T02:16:44.3289061Z 
2022-11-23T02:16:44.3289355Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3289499Z Ran 1 test in 1.637s
2022-11-23T02:16:44.3289518Z 
2022-11-23T02:16:44.3289727Z OK (skipped=1)
2022-11-23T02:16:44.3289764Z 
2022-11-23T02:16:44.3289906Z Generating XML reports...
2022-11-23T02:16:44.3290391Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020016.xml
2022-11-23T02:16:44.3290796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3291005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3291415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3291645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3291664Z 
2022-11-23T02:16:44.3291807Z Running tests...
2022-11-23T02:16:44.3292102Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3292433Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3292760Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3293536Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2022-11-23T02:16:44.3293556Z 
2022-11-23T02:16:44.3293848Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3294053Z Ran 1 test in 1.635s
2022-11-23T02:16:44.3294073Z 
2022-11-23T02:16:44.3294219Z OK (skipped=1)
2022-11-23T02:16:44.3294237Z 
2022-11-23T02:16:44.3294394Z Generating XML reports...
2022-11-23T02:16:44.3294876Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020020.xml
2022-11-23T02:16:44.3295283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3295493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3295892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3296116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3296136Z 
2022-11-23T02:16:44.3296275Z Running tests...
2022-11-23T02:16:44.3296570Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3296918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3297238Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3297490Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37701
2022-11-23T02:16:44.3297745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37702
2022-11-23T02:16:44.3298134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3298344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3298758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3298981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3299384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3299591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3299997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3300270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3300552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3300814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3301247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3301675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3301941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3302202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3302493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps51knp1w
2022-11-23T02:16:44.3302800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps51knp1w/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3303086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbz3njzin
2022-11-23T02:16:44.3303389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbz3njzin/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3303617Z <class 'torch.nn.parameter.Parameter'>
2022-11-23T02:16:44.3303754Z ok (5.128s)
2022-11-23T02:16:44.3303773Z 
2022-11-23T02:16:44.3304072Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3304219Z Ran 1 test in 5.128s
2022-11-23T02:16:44.3304238Z 
2022-11-23T02:16:44.3304410Z OK
2022-11-23T02:16:44.3304432Z 
2022-11-23T02:16:44.3304592Z Generating XML reports...
2022-11-23T02:16:44.3305082Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020024.xml
2022-11-23T02:16:44.3305488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3305687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3306101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3306327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3306346Z 
2022-11-23T02:16:44.3306488Z Running tests...
2022-11-23T02:16:44.3306783Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3307132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3307473Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3307727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37812
2022-11-23T02:16:44.3307960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37813
2022-11-23T02:16:44.3308371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3308580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3308992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3309215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3309613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3309818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3310223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3310445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3310763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3311040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3311477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3311908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3312174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3312437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3312724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnqi34wa
2022-11-23T02:16:44.3313026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnqi34wa/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3313316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n0femfi
2022-11-23T02:16:44.3313603Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n0femfi/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3313729Z ok (5.142s)
2022-11-23T02:16:44.3313748Z 
2022-11-23T02:16:44.3314051Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3314198Z Ran 1 test in 5.142s
2022-11-23T02:16:44.3314217Z 
2022-11-23T02:16:44.3314342Z OK
2022-11-23T02:16:44.3314361Z 
2022-11-23T02:16:44.3314516Z Generating XML reports...
2022-11-23T02:16:44.3315043Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020032.xml
2022-11-23T02:16:44.3315457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3315652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3316072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3316295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3316315Z 
2022-11-23T02:16:44.3316487Z Running tests...
2022-11-23T02:16:44.3316788Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3317134Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3317434Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3317686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37923
2022-11-23T02:16:44.3317933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37924
2022-11-23T02:16:44.3318320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3318532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3318946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3319168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3319561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3319765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3320174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3320392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3320653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3320984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3321422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3321851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3322115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3322376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3322671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4y4i_st
2022-11-23T02:16:44.3322972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4y4i_st/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3323258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdqzz38ye
2022-11-23T02:16:44.3323548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdqzz38ye/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3323819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3324081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3324219Z ok (5.671s)
2022-11-23T02:16:44.3324238Z 
2022-11-23T02:16:44.3324540Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3324686Z Ran 1 test in 5.672s
2022-11-23T02:16:44.3324705Z 
2022-11-23T02:16:44.3324831Z OK
2022-11-23T02:16:44.3324894Z 
2022-11-23T02:16:44.3325057Z Generating XML reports...
2022-11-23T02:16:44.3325531Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020039.xml
2022-11-23T02:16:44.3326190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3326414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3326830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3327055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3327075Z 
2022-11-23T02:16:44.3327216Z Running tests...
2022-11-23T02:16:44.3327513Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3327858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3328186Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3328422Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38038
2022-11-23T02:16:44.3328674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38039
2022-11-23T02:16:44.3329083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3329293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3329703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3329925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3330319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3330529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3330944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3331153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3331513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3331790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3332225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3332651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3332915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3333179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3333471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7qen__c
2022-11-23T02:16:44.3333756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7qen__c/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3334049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmqf7_haj
2022-11-23T02:16:44.3334349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmqf7_haj/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3335217Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:16:44.3335365Z ok (5.620s)
2022-11-23T02:16:44.3335385Z 
2022-11-23T02:16:44.3335688Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3335838Z Ran 1 test in 5.620s
2022-11-23T02:16:44.3335857Z 
2022-11-23T02:16:44.3335982Z OK
2022-11-23T02:16:44.3336000Z 
2022-11-23T02:16:44.3336156Z Generating XML reports...
2022-11-23T02:16:44.3336643Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020047.xml
2022-11-23T02:16:44.3337049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3337244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3337660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3337885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3337904Z 
2022-11-23T02:16:44.3338045Z Running tests...
2022-11-23T02:16:44.3338339Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3338694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3339011Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3339794Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2022-11-23T02:16:44.3339815Z 
2022-11-23T02:16:44.3340111Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3340241Z Ran 1 test in 1.635s
2022-11-23T02:16:44.3340275Z 
2022-11-23T02:16:44.3340401Z OK (skipped=1)
2022-11-23T02:16:44.3340420Z 
2022-11-23T02:16:44.3340576Z Generating XML reports...
2022-11-23T02:16:44.3341057Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020056.xml
2022-11-23T02:16:44.3341535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3341744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3342155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3342378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3342397Z 
2022-11-23T02:16:44.3342539Z Running tests...
2022-11-23T02:16:44.3342825Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3343171Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3343463Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3343714Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38187
2022-11-23T02:16:44.3343970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38188
2022-11-23T02:16:44.3344374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3344582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3344994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3345204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3345650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3345867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3346276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3346500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3346780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3347056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3347492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3347920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3348172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3348436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3348728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56xxvwf_
2022-11-23T02:16:44.3349032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56xxvwf_/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3349321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivsjj1cg
2022-11-23T02:16:44.3349624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivsjj1cg/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3350569Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:16:44.3351490Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:16:44.3352749Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T02:16:44.3353015Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-11-23T02:16:44.3354223Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T02:16:44.3354491Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-11-23T02:16:44.3354762Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3355077Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3356006Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:16:44.3356932Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:16:44.3357850Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:16:44.3358753Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:16:44.3359672Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:16:44.3360578Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:16:44.3361547Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:16:44.3362459Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:16:44.3363364Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:16:44.3364271Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:16:44.3364409Z ok (4.225s)
2022-11-23T02:16:44.3364429Z 
2022-11-23T02:16:44.3364769Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3364906Z Ran 1 test in 4.226s
2022-11-23T02:16:44.3364942Z 
2022-11-23T02:16:44.3365052Z OK
2022-11-23T02:16:44.3365071Z 
2022-11-23T02:16:44.3365231Z Generating XML reports...
2022-11-23T02:16:44.3365939Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020100.xml
2022-11-23T02:16:44.3366365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3366572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3366987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3367213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3367232Z 
2022-11-23T02:16:44.3367374Z Running tests...
2022-11-23T02:16:44.3367658Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3368007Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3368288Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3369068Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.652s)
2022-11-23T02:16:44.3369088Z 
2022-11-23T02:16:44.3369380Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3369525Z Ran 1 test in 1.652s
2022-11-23T02:16:44.3369544Z 
2022-11-23T02:16:44.3369685Z OK (skipped=1)
2022-11-23T02:16:44.3369704Z 
2022-11-23T02:16:44.3369860Z Generating XML reports...
2022-11-23T02:16:44.3370348Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020107.xml
2022-11-23T02:16:44.3370740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3370951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3371458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3371685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3371705Z 
2022-11-23T02:16:44.3371846Z Running tests...
2022-11-23T02:16:44.3372139Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3372486Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3372790Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3373045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38334
2022-11-23T02:16:44.3373278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38335
2022-11-23T02:16:44.3373683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3373898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3374311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3374535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3374932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3375136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3375600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3375817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3376100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3376381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3376816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3377246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3377507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3377767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3378058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazs95bp4
2022-11-23T02:16:44.3378363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazs95bp4/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3378633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp266_fr3t
2022-11-23T02:16:44.3378937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp266_fr3t/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3379761Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T02:16:44.3380124Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T02:16:44.3380944Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T02:16:44.3381363Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T02:16:44.3381499Z ok (5.548s)
2022-11-23T02:16:44.3381519Z 
2022-11-23T02:16:44.3381817Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3381964Z Ran 1 test in 5.548s
2022-11-23T02:16:44.3381983Z 
2022-11-23T02:16:44.3382108Z OK
2022-11-23T02:16:44.3382127Z 
2022-11-23T02:16:44.3382283Z Generating XML reports...
2022-11-23T02:16:44.3382751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020111.xml
2022-11-23T02:16:44.3383160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3383371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3383784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3384012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3384031Z 
2022-11-23T02:16:44.3384168Z Running tests...
2022-11-23T02:16:44.3384465Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3384811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3385101Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3385932Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.636s)
2022-11-23T02:16:44.3385972Z 
2022-11-23T02:16:44.3386264Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3386412Z Ran 1 test in 1.636s
2022-11-23T02:16:44.3386431Z 
2022-11-23T02:16:44.3386573Z OK (skipped=1)
2022-11-23T02:16:44.3386592Z 
2022-11-23T02:16:44.3386750Z Generating XML reports...
2022-11-23T02:16:44.3387234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020119.xml
2022-11-23T02:16:44.3387634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3387842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3388260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3388470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3388505Z 
2022-11-23T02:16:44.3388631Z Running tests...
2022-11-23T02:16:44.3388928Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3389280Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3389588Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3390363Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-11-23T02:16:44.3390383Z 
2022-11-23T02:16:44.3390677Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3390823Z Ran 1 test in 1.643s
2022-11-23T02:16:44.3390841Z 
2022-11-23T02:16:44.3390981Z OK (skipped=1)
2022-11-23T02:16:44.3391000Z 
2022-11-23T02:16:44.3391159Z Generating XML reports...
2022-11-23T02:16:44.3391628Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020123.xml
2022-11-23T02:16:44.3392091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3392298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3392708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3392930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3392949Z 
2022-11-23T02:16:44.3393091Z Running tests...
2022-11-23T02:16:44.3393387Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3393737Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3394045Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3394297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38547
2022-11-23T02:16:44.3394551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38548
2022-11-23T02:16:44.3394956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3395164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3395575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3395799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3396245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3396459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3396850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3397075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3397353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3397629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3398062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3398493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3398757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3399017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3399289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3399552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3399986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3400412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3400702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyckf4z7d
2022-11-23T02:16:44.3401008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyckf4z7d/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3401299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm25smlyq
2022-11-23T02:16:44.3401599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm25smlyq/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3401867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3402175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3402444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3402703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3402830Z ok (5.848s)
2022-11-23T02:16:44.3402850Z 
2022-11-23T02:16:44.3403153Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3403300Z Ran 1 test in 5.848s
2022-11-23T02:16:44.3403319Z 
2022-11-23T02:16:44.3403449Z OK
2022-11-23T02:16:44.3403468Z 
2022-11-23T02:16:44.3403624Z Generating XML reports...
2022-11-23T02:16:44.3404109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020127.xml
2022-11-23T02:16:44.3404501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3404715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3405129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3405353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3405372Z 
2022-11-23T02:16:44.3405515Z Running tests...
2022-11-23T02:16:44.3406028Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3406461Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3406780Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3407019Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38668
2022-11-23T02:16:44.3407271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38669
2022-11-23T02:16:44.3407680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3407888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3408302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3408524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3408922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3409132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3409548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3409754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3410036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3410311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3410746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3411172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3411435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3411699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3412010Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:16:44.3412317Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:16:44.3412675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp792n9q9
2022-11-23T02:16:44.3412978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp792n9q9/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3413265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpydojgghq
2022-11-23T02:16:44.3413566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpydojgghq/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3413833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3414105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3414368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3414635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3414936Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:16:44.3415244Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:16:44.3415550Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:16:44.3415851Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:16:44.3416191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3416507Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3416772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3417036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3417350Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:16:44.3417643Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:16:44.3417953Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T02:16:44.3418254Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T02:16:44.3418520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3418787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3419049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3419313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3419453Z ok (6.340s)
2022-11-23T02:16:44.3419472Z 
2022-11-23T02:16:44.3419779Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3419911Z Ran 1 test in 6.340s
2022-11-23T02:16:44.3419931Z 
2022-11-23T02:16:44.3420056Z OK
2022-11-23T02:16:44.3420075Z 
2022-11-23T02:16:44.3420231Z Generating XML reports...
2022-11-23T02:16:44.3420721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020136.xml
2022-11-23T02:16:44.3421129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3421339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3421749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3421974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3422042Z 
2022-11-23T02:16:44.3422187Z Running tests...
2022-11-23T02:16:44.3422473Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3422820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3423122Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3423908Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.636s)
2022-11-23T02:16:44.3423929Z 
2022-11-23T02:16:44.3424224Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3424370Z Ran 1 test in 1.636s
2022-11-23T02:16:44.3424389Z 
2022-11-23T02:16:44.3424531Z OK (skipped=1)
2022-11-23T02:16:44.3424550Z 
2022-11-23T02:16:44.3424707Z Generating XML reports...
2022-11-23T02:16:44.3425189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020144.xml
2022-11-23T02:16:44.3425576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3425787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3426197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3426469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3426490Z 
2022-11-23T02:16:44.3426635Z Running tests...
2022-11-23T02:16:44.3426933Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3427281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3427592Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3427844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38817
2022-11-23T02:16:44.3428079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38818
2022-11-23T02:16:44.3428486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3428696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3429112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3429339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3429733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3429941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3430350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3430558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3430839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3431117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3431554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3431983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3432246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3432558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3433140Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:16:44.3433720Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:16:44.3434013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe032r6ce
2022-11-23T02:16:44.3434315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe032r6ce/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3434593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdvzzfvj
2022-11-23T02:16:44.3434897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdvzzfvj/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3435165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3435432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3435783Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T02:16:44.3436096Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T02:16:44.3436427Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T02:16:44.3436762Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T02:16:44.3437123Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T02:16:44.3437488Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T02:16:44.3437836Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T02:16:44.3438187Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T02:16:44.3438454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3438724Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3438864Z ok (5.625s)
2022-11-23T02:16:44.3438884Z 
2022-11-23T02:16:44.3439190Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3439337Z Ran 1 test in 5.625s
2022-11-23T02:16:44.3439356Z 
2022-11-23T02:16:44.3439481Z OK
2022-11-23T02:16:44.3439499Z 
2022-11-23T02:16:44.3439640Z Generating XML reports...
2022-11-23T02:16:44.3440129Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020149.xml
2022-11-23T02:16:44.3440538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3440749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3441161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3441439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3441459Z 
2022-11-23T02:16:44.3441602Z Running tests...
2022-11-23T02:16:44.3441900Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3442246Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3442578Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3442831Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38932
2022-11-23T02:16:44.3443084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38933
2022-11-23T02:16:44.3443493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3443701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3444117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3444340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3444735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3444941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3445330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3445596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3446111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3446394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3446833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3447268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3447532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3447793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3448067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0_52gf6
2022-11-23T02:16:44.3448370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0_52gf6/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3448660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxrdi8vb
2022-11-23T02:16:44.3448960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxrdi8vb/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3449231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3449500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3449766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3450026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3450151Z ok (5.848s)
2022-11-23T02:16:44.3450171Z 
2022-11-23T02:16:44.3450457Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3450604Z Ran 1 test in 5.848s
2022-11-23T02:16:44.3450623Z 
2022-11-23T02:16:44.3450753Z OK
2022-11-23T02:16:44.3450772Z 
2022-11-23T02:16:44.3450928Z Generating XML reports...
2022-11-23T02:16:44.3451411Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020157.xml
2022-11-23T02:16:44.3451820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3452110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3452530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3452739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3452775Z 
2022-11-23T02:16:44.3452902Z Running tests...
2022-11-23T02:16:44.3453198Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3453548Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3453892Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3454144Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39077
2022-11-23T02:16:44.3454397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39078
2022-11-23T02:16:44.3454801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3455011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3455405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3455629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3456085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3456300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3456712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3456933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3457215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3457492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3457924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3458338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3458605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3458869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3459158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphcbfs4nh
2022-11-23T02:16:44.3459460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphcbfs4nh/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3459749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fo51l4_
2022-11-23T02:16:44.3460049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fo51l4_/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3460319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3460572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3460841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3461110Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3461247Z ok (5.775s)
2022-11-23T02:16:44.3461267Z 
2022-11-23T02:16:44.3461567Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3461714Z Ran 1 test in 5.775s
2022-11-23T02:16:44.3461779Z 
2022-11-23T02:16:44.3461907Z OK
2022-11-23T02:16:44.3461925Z 
2022-11-23T02:16:44.3462082Z Generating XML reports...
2022-11-23T02:16:44.3462575Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020205.xml
2022-11-23T02:16:44.3462963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3463173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3463584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3463812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3463831Z 
2022-11-23T02:16:44.3463973Z Running tests...
2022-11-23T02:16:44.3464268Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3464614Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3465025Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3465279Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39222
2022-11-23T02:16:44.3465513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39223
2022-11-23T02:16:44.3465916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3466169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3466587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3466810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3467203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3467413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3467817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3468024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3468301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3468579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3469011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3469438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3469703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3469964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3470253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp94if_cw
2022-11-23T02:16:44.3470554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp94if_cw/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3470827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzilfwqup
2022-11-23T02:16:44.3471131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzilfwqup/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3471400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3471667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3471934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3472254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3472390Z ok (5.869s)
2022-11-23T02:16:44.3472410Z 
2022-11-23T02:16:44.3472713Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3472843Z Ran 1 test in 5.869s
2022-11-23T02:16:44.3472878Z 
2022-11-23T02:16:44.3472987Z OK
2022-11-23T02:16:44.3473006Z 
2022-11-23T02:16:44.3473161Z Generating XML reports...
2022-11-23T02:16:44.3473649Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020214.xml
2022-11-23T02:16:44.3474059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3474267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3474678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3474907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3474926Z 
2022-11-23T02:16:44.3475066Z Running tests...
2022-11-23T02:16:44.3475346Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3475693Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3476098Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3476395Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39367
2022-11-23T02:16:44.3476652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39368
2022-11-23T02:16:44.3477054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3477265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3477673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3477897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3478275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3478480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3478886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3479109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3479387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3479661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3480097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3480523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3480771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3481033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3481328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ezpfj_b
2022-11-23T02:16:44.3481636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ezpfj_b/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3481920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_y8w5bk
2022-11-23T02:16:44.3482218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_y8w5bk/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3482551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3482818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3483085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3483335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3483473Z ok (5.731s)
2022-11-23T02:16:44.3483493Z 
2022-11-23T02:16:44.3483803Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3483949Z Ran 1 test in 5.732s
2022-11-23T02:16:44.3483968Z 
2022-11-23T02:16:44.3484094Z OK
2022-11-23T02:16:44.3484112Z 
2022-11-23T02:16:44.3484269Z Generating XML reports...
2022-11-23T02:16:44.3484752Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020222.xml
2022-11-23T02:16:44.3485162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3485357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3485979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3486219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3486239Z 
2022-11-23T02:16:44.3486383Z Running tests...
2022-11-23T02:16:44.3486751Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3487110Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3487514Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3487770Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39512
2022-11-23T02:16:44.3488020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39513
2022-11-23T02:16:44.3488412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3488623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3489034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3489262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3489659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3489865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3490269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3490497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3490776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3491039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3491476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3491907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3492168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3492429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3492784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4up87uru
2022-11-23T02:16:44.3493086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4up87uru/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3493369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4lqy82du
2022-11-23T02:16:44.3493670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4lqy82du/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3493923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3494196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3494463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3494727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3494862Z ok (5.750s)
2022-11-23T02:16:44.3494885Z 
2022-11-23T02:16:44.3495192Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3495338Z Ran 1 test in 5.751s
2022-11-23T02:16:44.3495358Z 
2022-11-23T02:16:44.3495485Z OK
2022-11-23T02:16:44.3495504Z 
2022-11-23T02:16:44.3495646Z Generating XML reports...
2022-11-23T02:16:44.3496129Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020230.xml
2022-11-23T02:16:44.3496529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3496780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3497202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3497423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3497444Z 
2022-11-23T02:16:44.3497586Z Running tests...
2022-11-23T02:16:44.3497884Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3498213Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3498615Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3498867Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39657
2022-11-23T02:16:44.3499116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39658
2022-11-23T02:16:44.3499519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3499728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3500138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3500369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3500770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3500960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3501365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3501586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3501868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3502149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3502579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3503063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3503372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3503632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3503906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5027w8ni
2022-11-23T02:16:44.3504205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5027w8ni/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3504493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_p6sc4w5
2022-11-23T02:16:44.3504790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_p6sc4w5/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3505057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3505329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3505594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3505861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3505981Z ok (5.756s)
2022-11-23T02:16:44.3506018Z 
2022-11-23T02:16:44.3506308Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3506454Z Ran 1 test in 5.756s
2022-11-23T02:16:44.3506474Z 
2022-11-23T02:16:44.3506598Z OK
2022-11-23T02:16:44.3506617Z 
2022-11-23T02:16:44.3506822Z Generating XML reports...
2022-11-23T02:16:44.3507322Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020239.xml
2022-11-23T02:16:44.3507721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3507933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3508348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3508558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3508593Z 
2022-11-23T02:16:44.3508720Z Running tests...
2022-11-23T02:16:44.3509016Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3509364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3509770Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3510025Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39802
2022-11-23T02:16:44.3510275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39803
2022-11-23T02:16:44.3510682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3510890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3511284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3511509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3511908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3512116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3512524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3512745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3513078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3513355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3513775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3514208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3514472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3514735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3515025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd81_6duj
2022-11-23T02:16:44.3515324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd81_6duj/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3515614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqc8ryliy
2022-11-23T02:16:44.3515916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqc8ryliy/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3516184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3516438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3516745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3517063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3517208Z ok (5.756s)
2022-11-23T02:16:44.3517227Z 
2022-11-23T02:16:44.3517527Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3517674Z Ran 1 test in 5.756s
2022-11-23T02:16:44.3517693Z 
2022-11-23T02:16:44.3517818Z OK
2022-11-23T02:16:44.3517841Z 
2022-11-23T02:16:44.3517998Z Generating XML reports...
2022-11-23T02:16:44.3518467Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020247.xml
2022-11-23T02:16:44.3518876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3519084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3519499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3519726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3519745Z 
2022-11-23T02:16:44.3519889Z Running tests...
2022-11-23T02:16:44.3520181Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3520530Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3520935Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3521169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39947
2022-11-23T02:16:44.3521421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39948
2022-11-23T02:16:44.3521827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3522034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3522447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3522671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3523065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3523328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3523731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3523935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3524213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3524488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3524924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3525351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3525613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3526092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3526389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5uakianj
2022-11-23T02:16:44.3526693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5uakianj/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3526964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1sultd3r
2022-11-23T02:16:44.3527265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1sultd3r/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3527606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3527883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3528150Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3528421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3528556Z ok (5.858s)
2022-11-23T02:16:44.3528576Z 
2022-11-23T02:16:44.3528885Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3529015Z Ran 1 test in 5.858s
2022-11-23T02:16:44.3529033Z 
2022-11-23T02:16:44.3529160Z OK
2022-11-23T02:16:44.3529178Z 
2022-11-23T02:16:44.3529330Z Generating XML reports...
2022-11-23T02:16:44.3529818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020255.xml
2022-11-23T02:16:44.3530225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3530435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3530848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3531076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3531096Z 
2022-11-23T02:16:44.3531239Z Running tests...
2022-11-23T02:16:44.3531518Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3531864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3532268Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3532526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40092
2022-11-23T02:16:44.3532776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40093
2022-11-23T02:16:44.3533181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3533386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3533871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3534080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3534476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3534683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3535092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3535314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3535593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3535868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3536305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3536738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3536985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3537247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3537538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3s4pr4a
2022-11-23T02:16:44.3537895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3s4pr4a/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3538189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1v592t_
2022-11-23T02:16:44.3538487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1v592t_/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3538757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3539024Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3539290Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3539537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3539663Z ok (5.816s)
2022-11-23T02:16:44.3539682Z 
2022-11-23T02:16:44.3540035Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3540235Z Ran 1 test in 5.817s
2022-11-23T02:16:44.3540255Z 
2022-11-23T02:16:44.3540417Z OK
2022-11-23T02:16:44.3540436Z 
2022-11-23T02:16:44.3540658Z Generating XML reports...
2022-11-23T02:16:44.3541304Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020304.xml
2022-11-23T02:16:44.3541708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3541949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3542402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3570605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3570638Z 
2022-11-23T02:16:44.3570804Z Running tests...
2022-11-23T02:16:44.3571150Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3571522Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3571929Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3572335Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40237
2022-11-23T02:16:44.3572584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40238
2022-11-23T02:16:44.3573013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3573220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3573645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3573869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3574280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3574485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3574905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3575130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3575406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3575678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3576126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3576633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3576904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3577163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3577451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpura5if5t
2022-11-23T02:16:44.3577750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpura5if5t/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3578033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzjj006ly
2022-11-23T02:16:44.3578331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzjj006ly/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3578595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3578851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3579119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3579385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3579503Z ok (5.710s)
2022-11-23T02:16:44.3579524Z 
2022-11-23T02:16:44.3579825Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3579972Z Ran 1 test in 5.710s
2022-11-23T02:16:44.3579992Z 
2022-11-23T02:16:44.3580110Z OK
2022-11-23T02:16:44.3580130Z 
2022-11-23T02:16:44.3580270Z Generating XML reports...
2022-11-23T02:16:44.3580772Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020312.xml
2022-11-23T02:16:44.3581189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3581388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3581811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3582028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3582049Z 
2022-11-23T02:16:44.3582170Z Running tests...
2022-11-23T02:16:44.3582474Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3582909Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3583262Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3583510Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40382
2022-11-23T02:16:44.3583763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40383
2022-11-23T02:16:44.3584182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3584391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3584820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3585034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3585452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3585660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3586087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3586308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3586587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3586909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3587363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3587799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3588066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3588331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3588622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6tqtdqa
2022-11-23T02:16:44.3588924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6tqtdqa/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3589211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02zvto3r
2022-11-23T02:16:44.3589515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02zvto3r/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3589786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3590052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3590320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3590588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3590705Z ok (5.844s)
2022-11-23T02:16:44.3590725Z 
2022-11-23T02:16:44.3591033Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3591168Z Ran 1 test in 5.844s
2022-11-23T02:16:44.3591187Z 
2022-11-23T02:16:44.3591304Z OK
2022-11-23T02:16:44.3591323Z 
2022-11-23T02:16:44.3591470Z Generating XML reports...
2022-11-23T02:16:44.3591971Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020320.xml
2022-11-23T02:16:44.3592379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3592583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3593063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3593282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3593302Z 
2022-11-23T02:16:44.3593435Z Running tests...
2022-11-23T02:16:44.3593734Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3594085Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3594434Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3594685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40527
2022-11-23T02:16:44.3594927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40528
2022-11-23T02:16:44.3595340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3595542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3595966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3596180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3596590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3596792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3597260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3597478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3597755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3598033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3598473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3598908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3599168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3599430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3599719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaquci_00
2022-11-23T02:16:44.3600016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaquci_00/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3600291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14iwvbb2
2022-11-23T02:16:44.3600593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14iwvbb2/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3600857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3601120Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3601378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3601638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3601767Z ok (5.758s)
2022-11-23T02:16:44.3601787Z 
2022-11-23T02:16:44.3602094Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3602224Z Ran 1 test in 5.759s
2022-11-23T02:16:44.3602245Z 
2022-11-23T02:16:44.3602347Z OK
2022-11-23T02:16:44.3602366Z 
2022-11-23T02:16:44.3602505Z Generating XML reports...
2022-11-23T02:16:44.3602999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020329.xml
2022-11-23T02:16:44.3603466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3603666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3604088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3604303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3604323Z 
2022-11-23T02:16:44.3604457Z Running tests...
2022-11-23T02:16:44.3604751Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3605101Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3605397Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3606436Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.593s)
2022-11-23T02:16:44.3606459Z 
2022-11-23T02:16:44.3606767Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3606898Z Ran 1 test in 1.593s
2022-11-23T02:16:44.3606918Z 
2022-11-23T02:16:44.3607047Z OK (skipped=1)
2022-11-23T02:16:44.3607066Z 
2022-11-23T02:16:44.3607206Z Generating XML reports...
2022-11-23T02:16:44.3607780Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020337.xml
2022-11-23T02:16:44.3608207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3608404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3608834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3609046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3609067Z 
2022-11-23T02:16:44.3609201Z Running tests...
2022-11-23T02:16:44.3609502Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3609854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3610142Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3610393Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40706
2022-11-23T02:16:44.3610635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40707
2022-11-23T02:16:44.3611049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3611255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3611679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3611895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3612307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3612504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3612928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3613148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3613418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3613760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3614203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3614635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3614896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3615157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3615446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_r0n6e8
2022-11-23T02:16:44.3615741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_r0n6e8/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3616024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6ysx899
2022-11-23T02:16:44.3616319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6ysx899/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3616437Z ok (5.835s)
2022-11-23T02:16:44.3616457Z 
2022-11-23T02:16:44.3616797Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3616925Z Ran 1 test in 5.835s
2022-11-23T02:16:44.3616945Z 
2022-11-23T02:16:44.3617062Z OK
2022-11-23T02:16:44.3617081Z 
2022-11-23T02:16:44.3617226Z Generating XML reports...
2022-11-23T02:16:44.3617719Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020341.xml
2022-11-23T02:16:44.3618181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3618381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3618809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3619030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3619050Z 
2022-11-23T02:16:44.3620154Z Running tests...
2022-11-23T02:16:44.3620460Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3620807Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3621111Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3621360Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40817
2022-11-23T02:16:44.3621613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40818
2022-11-23T02:16:44.3622024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3622233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3622666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3622888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3623301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3623505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3623929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3624153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3624431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3624704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3625217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3625656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3625918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3626183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3626473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0nkiw4tk
2022-11-23T02:16:44.3626779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0nkiw4tk/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3627066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9p_8wzi2
2022-11-23T02:16:44.3627361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9p_8wzi2/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3627635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3627904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3628348Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp0nkiw4tk'>
2022-11-23T02:16:44.3628541Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:16:44.3628976Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp9p_8wzi2'>
2022-11-23T02:16:44.3629168Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:16:44.3629338Z ok (5.656s)
2022-11-23T02:16:44.3629360Z 
2022-11-23T02:16:44.3629672Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3629798Z Ran 1 test in 5.656s
2022-11-23T02:16:44.3629817Z 
2022-11-23T02:16:44.3629934Z OK
2022-11-23T02:16:44.3629953Z 
2022-11-23T02:16:44.3630109Z Generating XML reports...
2022-11-23T02:16:44.3630607Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020350.xml
2022-11-23T02:16:44.3631022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3631229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3631654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3631877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3631901Z 
2022-11-23T02:16:44.3632032Z Running tests...
2022-11-23T02:16:44.3632339Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3632694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3632990Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3633249Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40932
2022-11-23T02:16:44.3633502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40933
2022-11-23T02:16:44.3633919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3634126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3634559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3634777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3635191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3635400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3635883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3636104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3636382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3636660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3637104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3637544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3637812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3638105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkshz805p
2022-11-23T02:16:44.3638411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkshz805p/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3638674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3638962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrgojqi8
2022-11-23T02:16:44.3639265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrgojqi8/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3639534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3639848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3639975Z ok (4.344s)
2022-11-23T02:16:44.3639994Z 
2022-11-23T02:16:44.3640301Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3640440Z Ran 1 test in 4.344s
2022-11-23T02:16:44.3640463Z 
2022-11-23T02:16:44.3640579Z OK
2022-11-23T02:16:44.3640598Z 
2022-11-23T02:16:44.3640745Z Generating XML reports...
2022-11-23T02:16:44.3641238Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020358.xml
2022-11-23T02:16:44.3641654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3641853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3642269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3642489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3642509Z 
2022-11-23T02:16:44.3642635Z Running tests...
2022-11-23T02:16:44.3642929Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3643277Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3643572Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3643821Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41075
2022-11-23T02:16:44.3644069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41076
2022-11-23T02:16:44.3644481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3644677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3645109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3645334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3646142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3646471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3646912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3647130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3647402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3647670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3648118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3648573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3648851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3649126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3649422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_duzls8g
2022-11-23T02:16:44.3649733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_duzls8g/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3650031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbiqk2m9p
2022-11-23T02:16:44.3650340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbiqk2m9p/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3650665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3650952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3651086Z ok (5.651s)
2022-11-23T02:16:44.3651106Z 
2022-11-23T02:16:44.3651423Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3651573Z Ran 1 test in 5.651s
2022-11-23T02:16:44.3651593Z 
2022-11-23T02:16:44.3651715Z OK
2022-11-23T02:16:44.3651734Z 
2022-11-23T02:16:44.3651890Z Generating XML reports...
2022-11-23T02:16:44.3652395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020405.xml
2022-11-23T02:16:44.3652819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3653017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3653459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3653689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3653709Z 
2022-11-23T02:16:44.3653851Z Running tests...
2022-11-23T02:16:44.3654161Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3654531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3654864Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3655123Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41190
2022-11-23T02:16:44.3655367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41191
2022-11-23T02:16:44.3655793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3656014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3656446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3656675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3657097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3657382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3657827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3658059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3658327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3658616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3659068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3659518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3659794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3660067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3660351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3660634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3661089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3661572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3661866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:16:44.3662155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:16:44.3662619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.3663067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.3663372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqnc7wjj
2022-11-23T02:16:44.3663688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqnc7wjj/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3663992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph17eq9do
2022-11-23T02:16:44.3664303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph17eq9do/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3664418Z ok (5.096s)
2022-11-23T02:16:44.3664438Z 
2022-11-23T02:16:44.3664752Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3664901Z Ran 1 test in 5.097s
2022-11-23T02:16:44.3664922Z 
2022-11-23T02:16:44.3665044Z OK
2022-11-23T02:16:44.3665063Z 
2022-11-23T02:16:44.3665221Z Generating XML reports...
2022-11-23T02:16:44.3665727Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020413.xml
2022-11-23T02:16:44.3666154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3666373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3666794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3667033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3667053Z 
2022-11-23T02:16:44.3667189Z Running tests...
2022-11-23T02:16:44.3667499Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3667930Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3668257Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3668520Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41313
2022-11-23T02:16:44.3668782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41314
2022-11-23T02:16:44.3669207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3669405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3669839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3670069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3670492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3670711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3671148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3671377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3671663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3671930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3672430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3672890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3673160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3673435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3673721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3674006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3674454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3674906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3675193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:16:44.3675462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:16:44.3675921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.3676369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.3676669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2nlx10dc
2022-11-23T02:16:44.3676987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2nlx10dc/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3677288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplia_xl2r
2022-11-23T02:16:44.3677602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplia_xl2r/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3677746Z ok (15.156s)
2022-11-23T02:16:44.3677766Z 
2022-11-23T02:16:44.3678055Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3678202Z Ran 1 test in 15.157s
2022-11-23T02:16:44.3678270Z 
2022-11-23T02:16:44.3678394Z OK
2022-11-23T02:16:44.3678414Z 
2022-11-23T02:16:44.3678570Z Generating XML reports...
2022-11-23T02:16:44.3679082Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020421.xml
2022-11-23T02:16:44.3679511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3679726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3680169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3680400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3680421Z 
2022-11-23T02:16:44.3680550Z Running tests...
2022-11-23T02:16:44.3680861Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3681228Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3681582Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3681842Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41436
2022-11-23T02:16:44.3682104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41437
2022-11-23T02:16:44.3682537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3682749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3683210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3683449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3683877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3684093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3684528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3684757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3685040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3685327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3685997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3686445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3686723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3687001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3687301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppiypkdp7
2022-11-23T02:16:44.3687617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppiypkdp7/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3687912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr54crzjf
2022-11-23T02:16:44.3688224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr54crzjf/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3688368Z ok (5.734s)
2022-11-23T02:16:44.3688388Z 
2022-11-23T02:16:44.3688706Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3688830Z Ran 1 test in 5.734s
2022-11-23T02:16:44.3688850Z 
2022-11-23T02:16:44.3688969Z OK
2022-11-23T02:16:44.3688988Z 
2022-11-23T02:16:44.3689223Z Generating XML reports...
2022-11-23T02:16:44.3689734Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020438.xml
2022-11-23T02:16:44.3690156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3690372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3690805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3691034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3691059Z 
2022-11-23T02:16:44.3691202Z Running tests...
2022-11-23T02:16:44.3691492Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3691856Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3692197Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3692458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41551
2022-11-23T02:16:44.3692716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41552
2022-11-23T02:16:44.3693142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3693357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3693890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3694114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3694541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3694759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3695201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3695432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3695719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3696003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3696457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3696913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3697172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3697442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3697740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vulsovd
2022-11-23T02:16:44.3698044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vulsovd/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3698335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5cbzbvh6
2022-11-23T02:16:44.3698631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5cbzbvh6/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3698759Z ok (5.782s)
2022-11-23T02:16:44.3698778Z 
2022-11-23T02:16:44.3699092Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3699219Z Ran 1 test in 5.782s
2022-11-23T02:16:44.3699239Z 
2022-11-23T02:16:44.3699363Z OK
2022-11-23T02:16:44.3699382Z 
2022-11-23T02:16:44.3699527Z Generating XML reports...
2022-11-23T02:16:44.3700036Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020447.xml
2022-11-23T02:16:44.3700519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3700739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3701179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3701415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3701434Z 
2022-11-23T02:16:44.3701572Z Running tests...
2022-11-23T02:16:44.3701867Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3702233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3702522Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3702780Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41666
2022-11-23T02:16:44.3703040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41667
2022-11-23T02:16:44.3703460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3703671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3704107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3704321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3704798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3705014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3705452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3705688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3705972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3706255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3706707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3707156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3707413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3707685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3707984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd6l5pymw
2022-11-23T02:16:44.3708282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmi4bm5b
2022-11-23T02:16:44.3708593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd6l5pymw/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3708900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmi4bm5b/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3709041Z ok (5.502s)
2022-11-23T02:16:44.3709061Z 
2022-11-23T02:16:44.3709374Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3709499Z Ran 1 test in 5.503s
2022-11-23T02:16:44.3709537Z 
2022-11-23T02:16:44.3709641Z OK
2022-11-23T02:16:44.3709664Z 
2022-11-23T02:16:44.3709818Z Generating XML reports...
2022-11-23T02:16:44.3710324Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020455.xml
2022-11-23T02:16:44.3710747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3711014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3711453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3711683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3711703Z 
2022-11-23T02:16:44.3711841Z Running tests...
2022-11-23T02:16:44.3712131Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3712498Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3712802Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3713065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41777
2022-11-23T02:16:44.3713323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41778
2022-11-23T02:16:44.3713755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3713970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3714397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3714608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3715027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3715290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3715731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3715962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3716247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3716582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3717041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3717494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3717750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3718027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3718326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeaf29_9d
2022-11-23T02:16:44.3718631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeaf29_9d/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3718925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6zs9cfuf
2022-11-23T02:16:44.3719234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6zs9cfuf/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3720120Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:16:44.3720989Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:16:44.3721326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3721602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3721732Z ok (5.508s)
2022-11-23T02:16:44.3721751Z 
2022-11-23T02:16:44.3722072Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3722198Z Ran 1 test in 5.508s
2022-11-23T02:16:44.3722237Z 
2022-11-23T02:16:44.3722336Z OK
2022-11-23T02:16:44.3722355Z 
2022-11-23T02:16:44.3722512Z Generating XML reports...
2022-11-23T02:16:44.3723022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020503.xml
2022-11-23T02:16:44.3723452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3723671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3724108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3724340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3724360Z 
2022-11-23T02:16:44.3724508Z Running tests...
2022-11-23T02:16:44.3724847Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3725221Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3725549Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3726563Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.632s)
2022-11-23T02:16:44.3726585Z 
2022-11-23T02:16:44.3726890Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3727038Z Ran 1 test in 1.632s
2022-11-23T02:16:44.3727058Z 
2022-11-23T02:16:44.3727196Z OK (skipped=1)
2022-11-23T02:16:44.3727215Z 
2022-11-23T02:16:44.3727373Z Generating XML reports...
2022-11-23T02:16:44.3727886Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020511.xml
2022-11-23T02:16:44.3728411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3728655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3729091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3729313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3729334Z 
2022-11-23T02:16:44.3729470Z Running tests...
2022-11-23T02:16:44.3729768Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3730120Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3730442Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3731227Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.646s)
2022-11-23T02:16:44.3731343Z 
2022-11-23T02:16:44.3731646Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3731770Z Ran 1 test in 1.647s
2022-11-23T02:16:44.3731810Z 
2022-11-23T02:16:44.3731935Z OK (skipped=1)
2022-11-23T02:16:44.3731954Z 
2022-11-23T02:16:44.3732106Z Generating XML reports...
2022-11-23T02:16:44.3732593Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020515.xml
2022-11-23T02:16:44.3733097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3733308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3733725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3733949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3733973Z 
2022-11-23T02:16:44.3734120Z Running tests...
2022-11-23T02:16:44.3734394Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3734739Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3735053Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3735300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41960
2022-11-23T02:16:44.3735548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41961
2022-11-23T02:16:44.3736016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3736230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3736644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3736850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3737253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3737457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3737860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3738081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3738364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3738639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3739070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3739512Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3739760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3740024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3740312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0or3tcjb
2022-11-23T02:16:44.3740615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0or3tcjb/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3740909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpziv1xp74
2022-11-23T02:16:44.3741211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpziv1xp74/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3741586Z STAGE:2022-11-23 02:05:24 41961:41961 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3741999Z STAGE:2022-11-23 02:05:24 41960:41960 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3742274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3742523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3742896Z STAGE:2022-11-23 02:05:25 41960:41960 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3743256Z STAGE:2022-11-23 02:05:25 41961:41961 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3743634Z STAGE:2022-11-23 02:05:25 41960:41960 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3744013Z STAGE:2022-11-23 02:05:25 41961:41961 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3744839Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:16:44.3745703Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:16:44.3746083Z STAGE:2022-11-23 02:05:25 41960:41960 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3746442Z STAGE:2022-11-23 02:05:25 41961:41961 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3746810Z STAGE:2022-11-23 02:05:25 41961:41961 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3747170Z STAGE:2022-11-23 02:05:25 41960:41960 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3747544Z STAGE:2022-11-23 02:05:25 41961:41961 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3747902Z STAGE:2022-11-23 02:05:25 41960:41960 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3748042Z ok (5.940s)
2022-11-23T02:16:44.3748062Z 
2022-11-23T02:16:44.3748358Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3748505Z Ran 1 test in 5.940s
2022-11-23T02:16:44.3748524Z 
2022-11-23T02:16:44.3748647Z OK
2022-11-23T02:16:44.3748665Z 
2022-11-23T02:16:44.3748817Z Generating XML reports...
2022-11-23T02:16:44.3749303Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020519.xml
2022-11-23T02:16:44.3749711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3749918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3750316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3750543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3750563Z 
2022-11-23T02:16:44.3750708Z Running tests...
2022-11-23T02:16:44.3750999Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3751345Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3751699Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3751951Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42079
2022-11-23T02:16:44.3752202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42080
2022-11-23T02:16:44.3752587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3752786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3753207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3753430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3753834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3754042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3754453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3754675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3754954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3755213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3755696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3756134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3756398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3756662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3756955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpld6x6y7v
2022-11-23T02:16:44.3757260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpld6x6y7v/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3757546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78wuyzqt
2022-11-23T02:16:44.3757846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78wuyzqt/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3757968Z ok (5.064s)
2022-11-23T02:16:44.3757988Z 
2022-11-23T02:16:44.3758290Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3758432Z Ran 1 test in 5.065s
2022-11-23T02:16:44.3758452Z 
2022-11-23T02:16:44.3758572Z OK
2022-11-23T02:16:44.3758591Z 
2022-11-23T02:16:44.3758743Z Generating XML reports...
2022-11-23T02:16:44.3759226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020528.xml
2022-11-23T02:16:44.3759635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3759840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3760236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3760458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3760478Z 
2022-11-23T02:16:44.3760620Z Running tests...
2022-11-23T02:16:44.3760918Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3761267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3761575Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3762409Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.596s)
2022-11-23T02:16:44.3762430Z 
2022-11-23T02:16:44.3762722Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3762862Z Ran 1 test in 1.596s
2022-11-23T02:16:44.3762882Z 
2022-11-23T02:16:44.3763016Z OK (skipped=1)
2022-11-23T02:16:44.3763034Z 
2022-11-23T02:16:44.3763164Z Generating XML reports...
2022-11-23T02:16:44.3763644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020535.xml
2022-11-23T02:16:44.3764042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3764246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3764666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3764889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3764908Z 
2022-11-23T02:16:44.3765052Z Running tests...
2022-11-23T02:16:44.3765346Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3765851Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3766256Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3766516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42224
2022-11-23T02:16:44.3766766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42225
2022-11-23T02:16:44.3767172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3767382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3767792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3768011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3768409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3768592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3769011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3769232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3769507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3769787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3770226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3770652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3770914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3771176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3771450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwl7nr3mp
2022-11-23T02:16:44.3771753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwl7nr3mp/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3772039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpos5yu94p
2022-11-23T02:16:44.3772405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpos5yu94p/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3773351Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:16:44.3773500Z   warnings.warn(
2022-11-23T02:16:44.3774448Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:16:44.3774597Z   warnings.warn(
2022-11-23T02:16:44.3774872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3775140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3775247Z ok (5.570s)
2022-11-23T02:16:44.3775267Z 
2022-11-23T02:16:44.3775566Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3775697Z Ran 1 test in 5.570s
2022-11-23T02:16:44.3775716Z 
2022-11-23T02:16:44.3775833Z OK
2022-11-23T02:16:44.3775851Z 
2022-11-23T02:16:44.3776004Z Generating XML reports...
2022-11-23T02:16:44.3776533Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020539.xml
2022-11-23T02:16:44.3776949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3777157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3777575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3777780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3777799Z 
2022-11-23T02:16:44.3777943Z Running tests...
2022-11-23T02:16:44.3778239Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3778584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3778895Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3779677Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2022-11-23T02:16:44.3779701Z 
2022-11-23T02:16:44.3779993Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3780136Z Ran 1 test in 1.633s
2022-11-23T02:16:44.3780155Z 
2022-11-23T02:16:44.3780300Z OK (skipped=1)
2022-11-23T02:16:44.3780319Z 
2022-11-23T02:16:44.3780448Z Generating XML reports...
2022-11-23T02:16:44.3780931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020548.xml
2022-11-23T02:16:44.3781333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3781538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3781947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3782166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3782234Z 
2022-11-23T02:16:44.3782372Z Running tests...
2022-11-23T02:16:44.3782666Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3783013Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3783302Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3783550Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42373
2022-11-23T02:16:44.3783801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42374
2022-11-23T02:16:44.3784210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3784414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3784825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3785049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3785449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3785634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3786046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3786264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3786586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3786871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3787313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3787749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3788011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3788269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3788539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64g69nwg
2022-11-23T02:16:44.3788839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64g69nwg/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3789129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vwy98n6
2022-11-23T02:16:44.3789420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vwy98n6/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3789790Z STAGE:2022-11-23 02:05:57 42373:42373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3790147Z STAGE:2022-11-23 02:05:57 42374:42374 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3790416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3790684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:16:44.3791271Z STAGE:2022-11-23 02:05:57 42374:42374 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:05:57 42373:42373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3791293Z 
2022-11-23T02:16:44.3791676Z STAGE:2022-11-23 02:05:57 42373:42373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3792041Z STAGE:2022-11-23 02:05:57 42374:42374 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3792396Z STAGE:2022-11-23 02:05:57 42373:42373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3792813Z STAGE:2022-11-23 02:05:57 42373:42373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3793188Z STAGE:2022-11-23 02:05:57 42373:42373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3793328Z ok (6.104s)
2022-11-23T02:16:44.3793348Z 
2022-11-23T02:16:44.3793639Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3793781Z Ran 1 test in 6.104s
2022-11-23T02:16:44.3793800Z 
2022-11-23T02:16:44.3793917Z OK
2022-11-23T02:16:44.3793936Z 
2022-11-23T02:16:44.3794072Z Generating XML reports...
2022-11-23T02:16:44.3794560Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020552.xml
2022-11-23T02:16:44.3794971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3795187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3795606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3795833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3795852Z 
2022-11-23T02:16:44.3795996Z Running tests...
2022-11-23T02:16:44.3796294Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3796642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3796970Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3797230Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42492
2022-11-23T02:16:44.3797484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42493
2022-11-23T02:16:44.3797896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3798110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3798524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3798749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3799148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3799339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3799756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3799981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3800259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3800541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3800980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3801412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3801681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3801945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3802223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1h1z8o8
2022-11-23T02:16:44.3802532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1h1z8o8/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3802820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyoizm2w
2022-11-23T02:16:44.3803189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyoizm2w/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3803375Z ok (5.122s)
2022-11-23T02:16:44.3803396Z 
2022-11-23T02:16:44.3803703Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3803850Z Ran 1 test in 5.122s
2022-11-23T02:16:44.3803869Z 
2022-11-23T02:16:44.3804000Z OK
2022-11-23T02:16:44.3804018Z 
2022-11-23T02:16:44.3804178Z Generating XML reports...
2022-11-23T02:16:44.3804646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020600.xml
2022-11-23T02:16:44.3805053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3805264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3805965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3806208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3806228Z 
2022-11-23T02:16:44.3806371Z Running tests...
2022-11-23T02:16:44.3806672Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3807024Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3807314Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3807569Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42603
2022-11-23T02:16:44.3807900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42604
2022-11-23T02:16:44.3808318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3808528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3808948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3809177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3809578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3809787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3810183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3810413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3810696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3810975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3811408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3811846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3812112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3812376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3812652Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7oue8lb
2022-11-23T02:16:44.3812964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7oue8lb/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3813255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrspuzjp
2022-11-23T02:16:44.3813563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrspuzjp/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3813756Z ok (5.024s)
2022-11-23T02:16:44.3813776Z 
2022-11-23T02:16:44.3814083Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3814234Z Ran 1 test in 5.024s
2022-11-23T02:16:44.3814253Z 
2022-11-23T02:16:44.3814382Z OK
2022-11-23T02:16:44.3814401Z 
2022-11-23T02:16:44.3814553Z Generating XML reports...
2022-11-23T02:16:44.3815020Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020608.xml
2022-11-23T02:16:44.3815429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3815643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3816062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3816289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3816313Z 
2022-11-23T02:16:44.3816459Z Running tests...
2022-11-23T02:16:44.3816800Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3817152Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3817444Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3818285Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.655s)
2022-11-23T02:16:44.3818329Z 
2022-11-23T02:16:44.3818614Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3818765Z Ran 1 test in 1.655s
2022-11-23T02:16:44.3818784Z 
2022-11-23T02:16:44.3818918Z OK (skipped=1)
2022-11-23T02:16:44.3818941Z 
2022-11-23T02:16:44.3819100Z Generating XML reports...
2022-11-23T02:16:44.3819586Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020616.xml
2022-11-23T02:16:44.3819989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3820200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3820613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3820823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3820861Z 
2022-11-23T02:16:44.3820988Z Running tests...
2022-11-23T02:16:44.3821287Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3821637Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3821937Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3822717Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.605s)
2022-11-23T02:16:44.3822738Z 
2022-11-23T02:16:44.3823033Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3823180Z Ran 1 test in 1.605s
2022-11-23T02:16:44.3823199Z 
2022-11-23T02:16:44.3823346Z OK (skipped=1)
2022-11-23T02:16:44.3823365Z 
2022-11-23T02:16:44.3823546Z Generating XML reports...
2022-11-23T02:16:44.3824014Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020620.xml
2022-11-23T02:16:44.3824421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3824686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3825104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3825329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3825348Z 
2022-11-23T02:16:44.3825485Z Running tests...
2022-11-23T02:16:44.3825784Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3826137Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3826444Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3827223Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.602s)
2022-11-23T02:16:44.3827266Z 
2022-11-23T02:16:44.3827542Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3827691Z Ran 1 test in 1.602s
2022-11-23T02:16:44.3827710Z 
2022-11-23T02:16:44.3827855Z OK (skipped=1)
2022-11-23T02:16:44.3827874Z 
2022-11-23T02:16:44.3828031Z Generating XML reports...
2022-11-23T02:16:44.3828514Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020624.xml
2022-11-23T02:16:44.3828961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3829182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3829601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3829812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3829852Z 
2022-11-23T02:16:44.3829977Z Running tests...
2022-11-23T02:16:44.3830272Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3830626Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3830958Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3831212Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42816
2022-11-23T02:16:44.3831467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42817
2022-11-23T02:16:44.3831880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3832092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3832494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3832722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3833126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3833335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3833747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3833975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3834256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3834536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3835016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3835447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3835716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3835981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3836271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnk_rj6gu
2022-11-23T02:16:44.3836578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnk_rj6gu/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3836867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6dyzced
2022-11-23T02:16:44.3837170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6dyzced/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3837313Z ok (5.504s)
2022-11-23T02:16:44.3837332Z 
2022-11-23T02:16:44.3837615Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3837754Z Ran 1 test in 5.504s
2022-11-23T02:16:44.3837774Z 
2022-11-23T02:16:44.3837900Z OK
2022-11-23T02:16:44.3837919Z 
2022-11-23T02:16:44.3838079Z Generating XML reports...
2022-11-23T02:16:44.3838572Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020628.xml
2022-11-23T02:16:44.3838976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3839232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3839659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3839866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3839909Z 
2022-11-23T02:16:44.3840026Z Running tests...
2022-11-23T02:16:44.3840329Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3840678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3840982Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3841234Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42931
2022-11-23T02:16:44.3841483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42932
2022-11-23T02:16:44.3841893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3842103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3842499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3842724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3843123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3843333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3843746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3843970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3844254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3844534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3844954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3845442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3845927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3846359Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T02:16:44.3846651Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T02:16:44.3846914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3847385Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T02:16:44.3847673Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T02:16:44.3847966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcf78xwgk
2022-11-23T02:16:44.3848252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcf78xwgk/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3848548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyod4jjwr
2022-11-23T02:16:44.3848851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyod4jjwr/_remote_module_non_scriptable.py
2022-11-23T02:16:44.3848989Z ok (5.143s)
2022-11-23T02:16:44.3849009Z 
2022-11-23T02:16:44.3849308Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3849456Z Ran 1 test in 5.143s
2022-11-23T02:16:44.3849475Z 
2022-11-23T02:16:44.3849603Z OK
2022-11-23T02:16:44.3849622Z 
2022-11-23T02:16:44.3849855Z Generating XML reports...
2022-11-23T02:16:44.3850350Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020636.xml
2022-11-23T02:16:44.3850738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3850957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3851373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3851596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3851615Z 
2022-11-23T02:16:44.3851752Z Running tests...
2022-11-23T02:16:44.3852050Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3852401Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3852699Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3852936Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43042
2022-11-23T02:16:44.3853188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43043
2022-11-23T02:16:44.3853587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3853801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3854209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3854438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3854838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3855046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3855465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3855675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3855956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3856352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3856791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3857224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3857493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3857758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3858034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3858312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3858725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3859156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3859285Z ok (4.461s)
2022-11-23T02:16:44.3859305Z 
2022-11-23T02:16:44.3859602Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3859746Z Ran 1 test in 4.461s
2022-11-23T02:16:44.3859765Z 
2022-11-23T02:16:44.3859896Z OK
2022-11-23T02:16:44.3859915Z 
2022-11-23T02:16:44.3860072Z Generating XML reports...
2022-11-23T02:16:44.3860603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020644.xml
2022-11-23T02:16:44.3860999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3861209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3861631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3861860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3861879Z 
2022-11-23T02:16:44.3862024Z Running tests...
2022-11-23T02:16:44.3862325Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3862674Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3862959Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3863222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43157
2022-11-23T02:16:44.3863457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43158
2022-11-23T02:16:44.3863862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3864076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3864492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3864715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3865119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3865330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3865746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3865952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3866229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3866509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3866994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3867429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3867695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3867963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3868241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3868519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3868924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3869357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3869486Z ok (4.356s)
2022-11-23T02:16:44.3869506Z 
2022-11-23T02:16:44.3869806Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3869955Z Ran 1 test in 4.356s
2022-11-23T02:16:44.3869974Z 
2022-11-23T02:16:44.3870101Z OK
2022-11-23T02:16:44.3870120Z 
2022-11-23T02:16:44.3870275Z Generating XML reports...
2022-11-23T02:16:44.3870761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020651.xml
2022-11-23T02:16:44.3871211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3871409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3871828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3872059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3872079Z 
2022-11-23T02:16:44.3872222Z Running tests...
2022-11-23T02:16:44.3872517Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3872862Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3873174Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3873960Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.664s)
2022-11-23T02:16:44.3873981Z 
2022-11-23T02:16:44.3874280Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3874412Z Ran 1 test in 1.664s
2022-11-23T02:16:44.3874448Z 
2022-11-23T02:16:44.3874563Z OK (skipped=1)
2022-11-23T02:16:44.3874582Z 
2022-11-23T02:16:44.3874743Z Generating XML reports...
2022-11-23T02:16:44.3875226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020658.xml
2022-11-23T02:16:44.3875629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3875838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3876256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3876483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3876503Z 
2022-11-23T02:16:44.3876638Z Running tests...
2022-11-23T02:16:44.3876914Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3877326Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3877636Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3878414Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.646s)
2022-11-23T02:16:44.3878435Z 
2022-11-23T02:16:44.3878732Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3878884Z Ran 1 test in 1.646s
2022-11-23T02:16:44.3878903Z 
2022-11-23T02:16:44.3879047Z OK (skipped=1)
2022-11-23T02:16:44.3879066Z 
2022-11-23T02:16:44.3879220Z Generating XML reports...
2022-11-23T02:16:44.3879704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020702.xml
2022-11-23T02:16:44.3880111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3880304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3880717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3880944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3880963Z 
2022-11-23T02:16:44.3881097Z Running tests...
2022-11-23T02:16:44.3881440Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3881800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3882108Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3882366Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43340
2022-11-23T02:16:44.3882600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43341
2022-11-23T02:16:44.3883006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3883218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3883631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3883861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3884261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3884470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3884883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3885111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3885374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3885654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3886332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3886771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3887037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3887302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3887442Z ok (4.317s)
2022-11-23T02:16:44.3887535Z 
2022-11-23T02:16:44.3887842Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3887971Z Ran 1 test in 4.317s
2022-11-23T02:16:44.3888011Z 
2022-11-23T02:16:44.3888122Z OK
2022-11-23T02:16:44.3888141Z 
2022-11-23T02:16:44.3888300Z Generating XML reports...
2022-11-23T02:16:44.3888790Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020706.xml
2022-11-23T02:16:44.3889191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3889405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3889819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3890043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3890063Z 
2022-11-23T02:16:44.3890204Z Running tests...
2022-11-23T02:16:44.3890481Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3890831Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3891106Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3891361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43449
2022-11-23T02:16:44.3891612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43450
2022-11-23T02:16:44.3892079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3892297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3892715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3892921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3893331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3893543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3893954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3894182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3894465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3894746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3895184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3895613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3895864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3896237Z STAGE:2022-11-23 02:07:17 43450:43450 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3896504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3896871Z STAGE:2022-11-23 02:07:17 43449:43449 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3897240Z STAGE:2022-11-23 02:07:17 43450:43450 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3897620Z STAGE:2022-11-23 02:07:17 43450:43450 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3897987Z STAGE:2022-11-23 02:07:17 43449:43449 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3898365Z STAGE:2022-11-23 02:07:17 43449:43449 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3898777Z STAGE:2022-11-23 02:07:17 43450:43450 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3899111Z STAGE:2022-11-23 02:07:17 43449:43449 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3899477Z STAGE:2022-11-23 02:07:17 43449:43449 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3899853Z STAGE:2022-11-23 02:07:17 43449:43449 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3900222Z STAGE:2022-11-23 02:07:17 43450:43450 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3900599Z STAGE:2022-11-23 02:07:17 43450:43450 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3900738Z ok (4.470s)
2022-11-23T02:16:44.3900758Z 
2022-11-23T02:16:44.3901057Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3901209Z Ran 1 test in 4.470s
2022-11-23T02:16:44.3901228Z 
2022-11-23T02:16:44.3901359Z OK
2022-11-23T02:16:44.3901377Z 
2022-11-23T02:16:44.3901516Z Generating XML reports...
2022-11-23T02:16:44.3902003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020713.xml
2022-11-23T02:16:44.3902410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3902621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3903081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3903313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3903332Z 
2022-11-23T02:16:44.3903468Z Running tests...
2022-11-23T02:16:44.3903770Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3904105Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3904397Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3904652Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43562
2022-11-23T02:16:44.3904904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43563
2022-11-23T02:16:44.3905308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3905519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3905932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3906156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3906555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3906749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3907161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3907385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3907663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3907939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3908381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3908813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3909077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3909393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3909513Z ok (4.196s)
2022-11-23T02:16:44.3909533Z 
2022-11-23T02:16:44.3909838Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3909984Z Ran 1 test in 4.196s
2022-11-23T02:16:44.3910003Z 
2022-11-23T02:16:44.3910133Z OK
2022-11-23T02:16:44.3910152Z 
2022-11-23T02:16:44.3910313Z Generating XML reports...
2022-11-23T02:16:44.3910799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020720.xml
2022-11-23T02:16:44.3911203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3911415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3911808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3912040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3912059Z 
2022-11-23T02:16:44.3912205Z Running tests...
2022-11-23T02:16:44.3912501Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3912850Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3913141Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T02:16:44.3913161Z 
2022-11-23T02:16:44.3913500Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3913654Z Ran 1 test in 0.002s
2022-11-23T02:16:44.3913674Z 
2022-11-23T02:16:44.3913819Z OK (skipped=1)
2022-11-23T02:16:44.3913838Z 
2022-11-23T02:16:44.3913979Z Generating XML reports...
2022-11-23T02:16:44.3914470Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020727.xml
2022-11-23T02:16:44.3914878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3915090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3915505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3915731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3915750Z 
2022-11-23T02:16:44.3915887Z Running tests...
2022-11-23T02:16:44.3916180Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3916509Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3916842Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3917103Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43704
2022-11-23T02:16:44.3917356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43705
2022-11-23T02:16:44.3917760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3917969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3918381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3918607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3919006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3919198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3919609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3919884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3920162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3920442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3920878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3921313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3921578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3921841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3922099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3922376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3922812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3923234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3923601Z STAGE:2022-11-23 02:07:33 43705:43705 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3924001Z STAGE:2022-11-23 02:07:33 43704:43704 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3924379Z STAGE:2022-11-23 02:07:33 43705:43705 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3924760Z STAGE:2022-11-23 02:07:33 43705:43705 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3925132Z STAGE:2022-11-23 02:07:33 43704:43704 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3925494Z STAGE:2022-11-23 02:07:33 43704:43704 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3926066Z STAGE:2022-11-23 02:07:33 43705:43705 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3926431Z STAGE:2022-11-23 02:07:33 43704:43704 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.3926801Z STAGE:2022-11-23 02:07:33 43704:43704 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3927185Z STAGE:2022-11-23 02:07:33 43704:43704 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3927549Z STAGE:2022-11-23 02:07:33 43705:43705 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.3927927Z STAGE:2022-11-23 02:07:33 43705:43705 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.3928070Z ok (4.210s)
2022-11-23T02:16:44.3928090Z 
2022-11-23T02:16:44.3928371Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3928522Z Ran 1 test in 4.210s
2022-11-23T02:16:44.3928541Z 
2022-11-23T02:16:44.3928671Z OK
2022-11-23T02:16:44.3928689Z 
2022-11-23T02:16:44.3928848Z Generating XML reports...
2022-11-23T02:16:44.3929336Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020729.xml
2022-11-23T02:16:44.3929744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3929955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3930371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3930596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3930688Z 
2022-11-23T02:16:44.3930819Z Running tests...
2022-11-23T02:16:44.3931122Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3931468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3931757Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3932012Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43823
2022-11-23T02:16:44.3932264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43824
2022-11-23T02:16:44.3932673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3932882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3933275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3933508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3933912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3934123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3934531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3934754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3935091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3935381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3935820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3936240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3936510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3936772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3936965Z skip: Skipped due to small world size. (4.344s)
2022-11-23T02:16:44.3936985Z 
2022-11-23T02:16:44.3937284Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3937432Z Ran 1 test in 4.344s
2022-11-23T02:16:44.3937451Z 
2022-11-23T02:16:44.3937597Z OK (skipped=1)
2022-11-23T02:16:44.3937616Z 
2022-11-23T02:16:44.3937776Z Generating XML reports...
2022-11-23T02:16:44.3938260Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020736.xml
2022-11-23T02:16:44.3938647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3938862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3939271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3939496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3939515Z 
2022-11-23T02:16:44.3939652Z Running tests...
2022-11-23T02:16:44.3939945Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3940298Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3940587Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3940823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43932
2022-11-23T02:16:44.3941074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43933
2022-11-23T02:16:44.3941531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3941741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3942152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3942376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3942778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3942992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3943407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3943614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3943896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3944175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3944613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3945041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3945309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3945623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3945768Z ok (4.156s)
2022-11-23T02:16:44.3945788Z 
2022-11-23T02:16:44.3946094Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3946226Z Ran 1 test in 4.156s
2022-11-23T02:16:44.3946248Z 
2022-11-23T02:16:44.3946376Z OK
2022-11-23T02:16:44.3946396Z 
2022-11-23T02:16:44.3946554Z Generating XML reports...
2022-11-23T02:16:44.3947041Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020743.xml
2022-11-23T02:16:44.3947445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3947655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3948071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3948297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3948317Z 
2022-11-23T02:16:44.3948444Z Running tests...
2022-11-23T02:16:44.3948740Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3949087Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3949392Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3950175Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.594s)
2022-11-23T02:16:44.3950196Z 
2022-11-23T02:16:44.3950489Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3950730Z Ran 1 test in 1.594s
2022-11-23T02:16:44.3950758Z 
2022-11-23T02:16:44.3950970Z OK (skipped=1)
2022-11-23T02:16:44.3950990Z 
2022-11-23T02:16:44.3951151Z Generating XML reports...
2022-11-23T02:16:44.3951649Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020749.xml
2022-11-23T02:16:44.3952114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3952327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3952741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3952965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3952984Z 
2022-11-23T02:16:44.3953129Z Running tests...
2022-11-23T02:16:44.3953428Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3953780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3954070Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3954307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44075
2022-11-23T02:16:44.3954563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44076
2022-11-23T02:16:44.3954970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3955182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3955595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3955820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3956268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3956485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3956899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3957107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3957387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3957665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3958099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3958531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3958799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3959066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3959339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3959600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3960032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3960454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3960593Z ok (4.245s)
2022-11-23T02:16:44.3960613Z 
2022-11-23T02:16:44.3960906Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3961055Z Ran 1 test in 4.246s
2022-11-23T02:16:44.3961074Z 
2022-11-23T02:16:44.3961210Z OK
2022-11-23T02:16:44.3961229Z 
2022-11-23T02:16:44.3961387Z Generating XML reports...
2022-11-23T02:16:44.3961877Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020753.xml
2022-11-23T02:16:44.3962266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3962526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3962944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3963170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3963190Z 
2022-11-23T02:16:44.3963331Z Running tests...
2022-11-23T02:16:44.3963630Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3963982Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3964266Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3964519Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44190
2022-11-23T02:16:44.3964754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44191
2022-11-23T02:16:44.3965161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3965369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3966088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3966326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3966735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3967026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3967455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3967663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3967951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3968234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3968672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3969106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3969373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3969642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3969783Z ok (4.274s)
2022-11-23T02:16:44.3969803Z 
2022-11-23T02:16:44.3970102Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3970233Z Ran 1 test in 4.274s
2022-11-23T02:16:44.3970257Z 
2022-11-23T02:16:44.3970389Z OK
2022-11-23T02:16:44.3970409Z 
2022-11-23T02:16:44.3970566Z Generating XML reports...
2022-11-23T02:16:44.3971053Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020800.xml
2022-11-23T02:16:44.3971457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3971667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3972081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3972311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3972331Z 
2022-11-23T02:16:44.3972475Z Running tests...
2022-11-23T02:16:44.3972753Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3973102Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3973453Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3973712Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44299
2022-11-23T02:16:44.3973965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44300
2022-11-23T02:16:44.3974374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3974586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3975001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3975209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3975612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3975828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3976246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3976469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3976749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3977029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3977517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3977962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3978210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3978478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3978616Z ok (4.446s)
2022-11-23T02:16:44.3978635Z 
2022-11-23T02:16:44.3978938Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3979083Z Ran 1 test in 4.446s
2022-11-23T02:16:44.3979102Z 
2022-11-23T02:16:44.3979231Z OK
2022-11-23T02:16:44.3979249Z 
2022-11-23T02:16:44.3979408Z Generating XML reports...
2022-11-23T02:16:44.3979897Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020807.xml
2022-11-23T02:16:44.3980289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3980502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3980917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3981146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3981166Z 
2022-11-23T02:16:44.3981308Z Running tests...
2022-11-23T02:16:44.3981606Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3981956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3982258Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3982510Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44408
2022-11-23T02:16:44.3982748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44409
2022-11-23T02:16:44.3983155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3983362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3983775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3984055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3984456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3984668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3985083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3985293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3985573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3985853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3986290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3986725Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3986989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3987253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3987528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3987845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3988267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3988692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3988832Z ok (4.246s)
2022-11-23T02:16:44.3988852Z 
2022-11-23T02:16:44.3989150Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3989299Z Ran 1 test in 4.246s
2022-11-23T02:16:44.3989318Z 
2022-11-23T02:16:44.3989445Z OK
2022-11-23T02:16:44.3989464Z 
2022-11-23T02:16:44.3989621Z Generating XML reports...
2022-11-23T02:16:44.3990103Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020814.xml
2022-11-23T02:16:44.3990515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3990708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3991125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3991349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3991372Z 
2022-11-23T02:16:44.3991508Z Running tests...
2022-11-23T02:16:44.3991805Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3992154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.3992450Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.3992705Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44523
2022-11-23T02:16:44.3992938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44524
2022-11-23T02:16:44.3993347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3993558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3993968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3994248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3994651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.3994861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.3995271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.3995496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.3995761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.3996041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.3996477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3996915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.3997180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.3997440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.3997713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.3997988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.3998458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3998875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.3999014Z ok (4.327s)
2022-11-23T02:16:44.3999037Z 
2022-11-23T02:16:44.3999341Z ----------------------------------------------------------------------
2022-11-23T02:16:44.3999487Z Ran 1 test in 4.327s
2022-11-23T02:16:44.3999506Z 
2022-11-23T02:16:44.3999635Z OK
2022-11-23T02:16:44.3999654Z 
2022-11-23T02:16:44.3999812Z Generating XML reports...
2022-11-23T02:16:44.4000296Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020821.xml
2022-11-23T02:16:44.4000696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4000893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4001306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4001532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4001551Z 
2022-11-23T02:16:44.4001683Z Running tests...
2022-11-23T02:16:44.4001983Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4002330Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4002626Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4002880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44638
2022-11-23T02:16:44.4003130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44639
2022-11-23T02:16:44.4003521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4003732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4004141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4004367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4004828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4005042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4005454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4005900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4006171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4006451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4006894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4007328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4007594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4007857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4008153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpunnv2tmb
2022-11-23T02:16:44.4008459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpunnv2tmb/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4008749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34nhr_76
2022-11-23T02:16:44.4009107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34nhr_76/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4009255Z ok (5.634s)
2022-11-23T02:16:44.4009275Z 
2022-11-23T02:16:44.4009580Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4009723Z Ran 1 test in 5.634s
2022-11-23T02:16:44.4009746Z 
2022-11-23T02:16:44.4009876Z OK
2022-11-23T02:16:44.4009895Z 
2022-11-23T02:16:44.4010049Z Generating XML reports...
2022-11-23T02:16:44.4010536Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020828.xml
2022-11-23T02:16:44.4010940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4011148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4011549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4011779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4011798Z 
2022-11-23T02:16:44.4011944Z Running tests...
2022-11-23T02:16:44.4012241Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4012586Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4012862Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4013115Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44753
2022-11-23T02:16:44.4013365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44754
2022-11-23T02:16:44.4013754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4013964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4014376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4014600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4014999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4015274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4015687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4015910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4016173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4016453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4016936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4017372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4017637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4017907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4018044Z ok (4.112s)
2022-11-23T02:16:44.4018064Z 
2022-11-23T02:16:44.4018364Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4018513Z Ran 1 test in 4.112s
2022-11-23T02:16:44.4018534Z 
2022-11-23T02:16:44.4018643Z OK
2022-11-23T02:16:44.4018662Z 
2022-11-23T02:16:44.4018822Z Generating XML reports...
2022-11-23T02:16:44.4019305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020836.xml
2022-11-23T02:16:44.4019793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4020009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4020426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4020656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4020675Z 
2022-11-23T02:16:44.4020821Z Running tests...
2022-11-23T02:16:44.4021120Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4021450Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4021723Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4021978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44862
2022-11-23T02:16:44.4022233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44863
2022-11-23T02:16:44.4022641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4022848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4023264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4023488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4023869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4024079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4024491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4024716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4024997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4025274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4025710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4026212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4026478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4026722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4026864Z ok (4.444s)
2022-11-23T02:16:44.4026883Z 
2022-11-23T02:16:44.4027187Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4027340Z Ran 1 test in 4.444s
2022-11-23T02:16:44.4027360Z 
2022-11-23T02:16:44.4027491Z OK
2022-11-23T02:16:44.4027510Z 
2022-11-23T02:16:44.4027667Z Generating XML reports...
2022-11-23T02:16:44.4028155Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020843.xml
2022-11-23T02:16:44.4028560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4028753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4029166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4029393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4029412Z 
2022-11-23T02:16:44.4029557Z Running tests...
2022-11-23T02:16:44.4029850Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4030242Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4030554Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4030807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44971
2022-11-23T02:16:44.4031063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44972
2022-11-23T02:16:44.4031451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4031661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4032074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4032296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4032702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4032910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4033321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4033547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4033811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4034090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4034527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4034958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4035228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4035600Z STAGE:2022-11-23 02:08:53 44972:44972 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4035863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4036234Z STAGE:2022-11-23 02:08:53 44971:44971 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4036660Z STAGE:2022-11-23 02:08:53 44971:44971 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4037019Z STAGE:2022-11-23 02:08:53 44971:44971 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4037391Z STAGE:2022-11-23 02:08:53 44972:44972 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4037769Z STAGE:2022-11-23 02:08:53 44972:44972 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4037909Z ok (4.303s)
2022-11-23T02:16:44.4037932Z 
2022-11-23T02:16:44.4038232Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4038379Z Ran 1 test in 4.303s
2022-11-23T02:16:44.4038399Z 
2022-11-23T02:16:44.4038527Z OK
2022-11-23T02:16:44.4038545Z 
2022-11-23T02:16:44.4038702Z Generating XML reports...
2022-11-23T02:16:44.4039195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020849.xml
2022-11-23T02:16:44.4039586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4039794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4040208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4040467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4040487Z 
2022-11-23T02:16:44.4040664Z Running tests...
2022-11-23T02:16:44.4040967Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4041318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4041617Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4041859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45084
2022-11-23T02:16:44.4042115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45085
2022-11-23T02:16:44.4042516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4042726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4043137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4043364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4043766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4043975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4044388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4044598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4044876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4045154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4045589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4046249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4046519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4046889Z STAGE:2022-11-23 02:09:00 45085:45085 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4047233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4047606Z STAGE:2022-11-23 02:09:00 45084:45084 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4047957Z STAGE:2022-11-23 02:09:00 45084:45084 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4048340Z STAGE:2022-11-23 02:09:00 45084:45084 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4048712Z STAGE:2022-11-23 02:09:00 45085:45085 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4049092Z STAGE:2022-11-23 02:09:00 45085:45085 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4049229Z ok (4.472s)
2022-11-23T02:16:44.4049249Z 
2022-11-23T02:16:44.4049547Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4049692Z Ran 1 test in 4.472s
2022-11-23T02:16:44.4049712Z 
2022-11-23T02:16:44.4049844Z OK
2022-11-23T02:16:44.4049862Z 
2022-11-23T02:16:44.4050003Z Generating XML reports...
2022-11-23T02:16:44.4050494Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020856.xml
2022-11-23T02:16:44.4050898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4051108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4051519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4051802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4051823Z 
2022-11-23T02:16:44.4052025Z Running tests...
2022-11-23T02:16:44.4052327Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4052675Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4052981Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4053237Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45197
2022-11-23T02:16:44.4053490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45198
2022-11-23T02:16:44.4053896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4054104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4054517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4054745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4055146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4055342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4055757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4055982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4056262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4056541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4056978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4057410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4057675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4057999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4058257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.4058535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.4058971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4059397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4059674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:16:44.4059951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:16:44.4060370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.4060797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.4061063Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:16:44.4061172Z ok (21.738s)
2022-11-23T02:16:44.4061210Z 
2022-11-23T02:16:44.4061494Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4061644Z Ran 1 test in 21.738s
2022-11-23T02:16:44.4061663Z 
2022-11-23T02:16:44.4061791Z OK
2022-11-23T02:16:44.4061810Z 
2022-11-23T02:16:44.4062012Z Generating XML reports...
2022-11-23T02:16:44.4062506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020903.xml
2022-11-23T02:16:44.4062912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4063128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4063545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4063752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4063772Z 
2022-11-23T02:16:44.4063908Z Running tests...
2022-11-23T02:16:44.4064205Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4064553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4064893Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4065147Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45324
2022-11-23T02:16:44.4065399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45325
2022-11-23T02:16:44.4065808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4065999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4066412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4066642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4067042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4067257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4067669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4067894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4068172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4068505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4068923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4069352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4069621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4069889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4070165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.4070437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.4070869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4071288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4071560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:16:44.4071816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:16:44.4072285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.4072716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.4072984Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:16:44.4073245Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:16:44.4073390Z ok (21.656s)
2022-11-23T02:16:44.4073410Z 
2022-11-23T02:16:44.4073712Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4073858Z Ran 1 test in 21.657s
2022-11-23T02:16:44.4073877Z 
2022-11-23T02:16:44.4074005Z OK
2022-11-23T02:16:44.4074024Z 
2022-11-23T02:16:44.4074162Z Generating XML reports...
2022-11-23T02:16:44.4074649Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020928.xml
2022-11-23T02:16:44.4075056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4075266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4075683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4075908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4075931Z 
2022-11-23T02:16:44.4076066Z Running tests...
2022-11-23T02:16:44.4076361Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4076691Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4077007Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4077263Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45451
2022-11-23T02:16:44.4077515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45452
2022-11-23T02:16:44.4077921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4078131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4078547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4078831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4079237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4079428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4079841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4080065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4080347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4080626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4081058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4081496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4081761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4082025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4082201Z skip: Skipped due to small world size. (4.111s)
2022-11-23T02:16:44.4082221Z 
2022-11-23T02:16:44.4082520Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4082713Z Ran 1 test in 4.111s
2022-11-23T02:16:44.4082733Z 
2022-11-23T02:16:44.4082879Z OK (skipped=1)
2022-11-23T02:16:44.4082898Z 
2022-11-23T02:16:44.4083057Z Generating XML reports...
2022-11-23T02:16:44.4083546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020952.xml
2022-11-23T02:16:44.4083953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4084162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4084558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4084783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4084803Z 
2022-11-23T02:16:44.4084939Z Running tests...
2022-11-23T02:16:44.4085233Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4085585Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4086102Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4086366Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45560
2022-11-23T02:16:44.4086623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45561
2022-11-23T02:16:44.4087036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4087227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4087642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4087868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4088273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4088485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4088897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4089120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4089483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4089746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4090185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4090616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4090884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4091151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4091417Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-11-23T02:16:44.4091552Z ok (6.322s)
2022-11-23T02:16:44.4091576Z 
2022-11-23T02:16:44.4091877Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4092026Z Ran 1 test in 6.322s
2022-11-23T02:16:44.4092046Z 
2022-11-23T02:16:44.4092158Z OK
2022-11-23T02:16:44.4092177Z 
2022-11-23T02:16:44.4092332Z Generating XML reports...
2022-11-23T02:16:44.4092818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020958.xml
2022-11-23T02:16:44.4093217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4093487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4093912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4094140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4094159Z 
2022-11-23T02:16:44.4094309Z Running tests...
2022-11-23T02:16:44.4094610Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4094940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4095258Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4095514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45669
2022-11-23T02:16:44.4095767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45670
2022-11-23T02:16:44.4096179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4096390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4096803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4097034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4097415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4097622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4098035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4098260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4098542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4098825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4099260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4099690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4100020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4100264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4100544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.4100819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.4101257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4101683Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4101936Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-11-23T02:16:44.4102152Z No ranks successfully processed in monitoredBarrier.
2022-11-23T02:16:44.4102411Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-11-23T02:16:44.4102533Z ok (4.256s)
2022-11-23T02:16:44.4102571Z 
2022-11-23T02:16:44.4102855Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4103002Z Ran 1 test in 4.256s
2022-11-23T02:16:44.4103021Z 
2022-11-23T02:16:44.4103149Z OK
2022-11-23T02:16:44.4103168Z 
2022-11-23T02:16:44.4103371Z Generating XML reports...
2022-11-23T02:16:44.4103916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021007.xml
2022-11-23T02:16:44.4104333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4104545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4104961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4105172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4105211Z 
2022-11-23T02:16:44.4105339Z Running tests...
2022-11-23T02:16:44.4105639Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4105985Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4106300Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4106558Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45784
2022-11-23T02:16:44.4106812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45785
2022-11-23T02:16:44.4107216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4107424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4107825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4108050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4108455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4108666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4109079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4109305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4109585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4110023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4110338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4110775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4111042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4111305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4111582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.4111859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.4112295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4112720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4112990Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:16:44.4113111Z ok (4.514s)
2022-11-23T02:16:44.4113131Z 
2022-11-23T02:16:44.4113431Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4113582Z Ran 1 test in 4.515s
2022-11-23T02:16:44.4113602Z 
2022-11-23T02:16:44.4113729Z OK
2022-11-23T02:16:44.4113748Z 
2022-11-23T02:16:44.4113904Z Generating XML reports...
2022-11-23T02:16:44.4114433Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021014.xml
2022-11-23T02:16:44.4114847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4115059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4115470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4115680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4115700Z 
2022-11-23T02:16:44.4115834Z Running tests...
2022-11-23T02:16:44.4116133Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4116481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4116835Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4117097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45899
2022-11-23T02:16:44.4117350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45900
2022-11-23T02:16:44.4117754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4117951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4118353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4118565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4118975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4119197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4119610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4119835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4120113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4120393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4120868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4121299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4121566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4121832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4122023Z skip: Skipped due to small world size. (4.243s)
2022-11-23T02:16:44.4122046Z 
2022-11-23T02:16:44.4122346Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4122494Z Ran 1 test in 4.243s
2022-11-23T02:16:44.4122513Z 
2022-11-23T02:16:44.4122646Z OK (skipped=1)
2022-11-23T02:16:44.4122664Z 
2022-11-23T02:16:44.4122821Z Generating XML reports...
2022-11-23T02:16:44.4123294Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021021.xml
2022-11-23T02:16:44.4123699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4123909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4124318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4124547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4124566Z 
2022-11-23T02:16:44.4124748Z Running tests...
2022-11-23T02:16:44.4125052Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4125402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4126037Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T02:16:44.4126085Z 
2022-11-23T02:16:44.4126367Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4126516Z Ran 1 test in 0.003s
2022-11-23T02:16:44.4126536Z 
2022-11-23T02:16:44.4126682Z OK (skipped=1)
2022-11-23T02:16:44.4126701Z 
2022-11-23T02:16:44.4126858Z Generating XML reports...
2022-11-23T02:16:44.4127341Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021028.xml
2022-11-23T02:16:44.4127750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4127962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4128376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4128583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4128624Z 
2022-11-23T02:16:44.4128751Z Running tests...
2022-11-23T02:16:44.4129045Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4129394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4129832Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T02:16:44.4129852Z 
2022-11-23T02:16:44.4130144Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4130290Z Ran 1 test in 0.003s
2022-11-23T02:16:44.4130313Z 
2022-11-23T02:16:44.4130448Z OK (skipped=1)
2022-11-23T02:16:44.4130467Z 
2022-11-23T02:16:44.4130625Z Generating XML reports...
2022-11-23T02:16:44.4131091Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021030.xml
2022-11-23T02:16:44.4131571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4131781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4132196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4132423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4132443Z 
2022-11-23T02:16:44.4132591Z Running tests...
2022-11-23T02:16:44.4132884Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4133236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4133655Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T02:16:44.4133695Z 
2022-11-23T02:16:44.4133969Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4134119Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4134138Z 
2022-11-23T02:16:44.4134271Z OK (skipped=1)
2022-11-23T02:16:44.4134290Z 
2022-11-23T02:16:44.4134447Z Generating XML reports...
2022-11-23T02:16:44.4134927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021033.xml
2022-11-23T02:16:44.4135332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4135539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4136015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4136232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4136274Z 
2022-11-23T02:16:44.4136389Z Running tests...
2022-11-23T02:16:44.4136690Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4137042Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4137473Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-11-23T02:16:44.4137492Z 
2022-11-23T02:16:44.4137782Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4137929Z Ran 1 test in 0.003s
2022-11-23T02:16:44.4137948Z 
2022-11-23T02:16:44.4138093Z OK (skipped=1)
2022-11-23T02:16:44.4138112Z 
2022-11-23T02:16:44.4138271Z Generating XML reports...
2022-11-23T02:16:44.4138737Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021035.xml
2022-11-23T02:16:44.4139141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4139352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4139768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4139990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4140009Z 
2022-11-23T02:16:44.4140156Z Running tests...
2022-11-23T02:16:44.4140446Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4140791Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4141129Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-11-23T02:16:44.4141149Z 
2022-11-23T02:16:44.4141425Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4141574Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4141593Z 
2022-11-23T02:16:44.4141735Z OK (skipped=1)
2022-11-23T02:16:44.4141754Z 
2022-11-23T02:16:44.4141965Z Generating XML reports...
2022-11-23T02:16:44.4142455Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021038.xml
2022-11-23T02:16:44.4142860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4143072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4143483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4143712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4143731Z 
2022-11-23T02:16:44.4143849Z Running tests...
2022-11-23T02:16:44.4144145Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4144493Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4144777Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:16:44.4144800Z 
2022-11-23T02:16:44.4145096Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4145242Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4145261Z 
2022-11-23T02:16:44.4145394Z OK (skipped=1)
2022-11-23T02:16:44.4145413Z 
2022-11-23T02:16:44.4145569Z Generating XML reports...
2022-11-23T02:16:44.4146031Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021040.xml
2022-11-23T02:16:44.4146487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4146705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4147122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4147349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4147372Z 
2022-11-23T02:16:44.4147506Z Running tests...
2022-11-23T02:16:44.4147801Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4148150Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4148456Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:16:44.4148476Z 
2022-11-23T02:16:44.4148753Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4148900Z Ran 1 test in 0.003s
2022-11-23T02:16:44.4148922Z 
2022-11-23T02:16:44.4149055Z OK (skipped=1)
2022-11-23T02:16:44.4149074Z 
2022-11-23T02:16:44.4149230Z Generating XML reports...
2022-11-23T02:16:44.4149711Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021042.xml
2022-11-23T02:16:44.4150119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4150329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4150745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4150970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4150990Z 
2022-11-23T02:16:44.4151105Z Running tests...
2022-11-23T02:16:44.4151401Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4151752Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4152095Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:16:44.4152115Z 
2022-11-23T02:16:44.4152413Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4152616Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4152635Z 
2022-11-23T02:16:44.4152780Z OK (skipped=1)
2022-11-23T02:16:44.4152799Z 
2022-11-23T02:16:44.4152959Z Generating XML reports...
2022-11-23T02:16:44.4153445Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021045.xml
2022-11-23T02:16:44.4153832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4154043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4154456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4154680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4154700Z 
2022-11-23T02:16:44.4154844Z Running tests...
2022-11-23T02:16:44.4155137Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4155489Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4155829Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4156068Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46272
2022-11-23T02:16:44.4156318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46273
2022-11-23T02:16:44.4156762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4156981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4157395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4157621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4158030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4158242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4158653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4158860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4159140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4159423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4159859Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4160288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4160559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4160826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4160962Z ok (4.224s)
2022-11-23T02:16:44.4160982Z 
2022-11-23T02:16:44.4161281Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4161401Z Ran 1 test in 4.225s
2022-11-23T02:16:44.4161420Z 
2022-11-23T02:16:44.4161549Z OK
2022-11-23T02:16:44.4161567Z 
2022-11-23T02:16:44.4161724Z Generating XML reports...
2022-11-23T02:16:44.4162209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021047.xml
2022-11-23T02:16:44.4162611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4162822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4163291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4163519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4163538Z 
2022-11-23T02:16:44.4163665Z Running tests...
2022-11-23T02:16:44.4163963Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4164311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4164644Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4164900Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46381
2022-11-23T02:16:44.4165150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46382
2022-11-23T02:16:44.4165552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4165980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4166407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4166613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4167018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4167228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4167708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4167939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4168222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4168507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4168945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4169355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4169623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4169882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4170023Z ok (4.153s)
2022-11-23T02:16:44.4170044Z 
2022-11-23T02:16:44.4170343Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4170491Z Ran 1 test in 4.154s
2022-11-23T02:16:44.4170510Z 
2022-11-23T02:16:44.4170638Z OK
2022-11-23T02:16:44.4170657Z 
2022-11-23T02:16:44.4170814Z Generating XML reports...
2022-11-23T02:16:44.4171305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021054.xml
2022-11-23T02:16:44.4171688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4171899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4172311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4172536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4172560Z 
2022-11-23T02:16:44.4172705Z Running tests...
2022-11-23T02:16:44.4173002Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4173350Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4173663Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:16:44.4173762Z 
2022-11-23T02:16:44.4174060Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4174179Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4174198Z 
2022-11-23T02:16:44.4174341Z OK (skipped=1)
2022-11-23T02:16:44.4174360Z 
2022-11-23T02:16:44.4174506Z Generating XML reports...
2022-11-23T02:16:44.4174986Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021101.xml
2022-11-23T02:16:44.4175389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4175600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4176015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4176238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4176261Z 
2022-11-23T02:16:44.4176388Z Running tests...
2022-11-23T02:16:44.4176683Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4177028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4177363Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:16:44.4177383Z 
2022-11-23T02:16:44.4177678Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4177871Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4177892Z 
2022-11-23T02:16:44.4178034Z OK (skipped=1)
2022-11-23T02:16:44.4178053Z 
2022-11-23T02:16:44.4178211Z Generating XML reports...
2022-11-23T02:16:44.4178696Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021103.xml
2022-11-23T02:16:44.4179087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4179299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4179716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4179942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4179961Z 
2022-11-23T02:16:44.4180102Z Running tests...
2022-11-23T02:16:44.4180400Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4180752Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4181069Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4181858Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.619s)
2022-11-23T02:16:44.4181883Z 
2022-11-23T02:16:44.4182181Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4182309Z Ran 1 test in 1.619s
2022-11-23T02:16:44.4182329Z 
2022-11-23T02:16:44.4182474Z OK (skipped=1)
2022-11-23T02:16:44.4182492Z 
2022-11-23T02:16:44.4182651Z Generating XML reports...
2022-11-23T02:16:44.4183136Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021105.xml
2022-11-23T02:16:44.4183542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4183759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4184169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4184486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4184505Z 
2022-11-23T02:16:44.4184652Z Running tests...
2022-11-23T02:16:44.4184930Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4185276Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4185593Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4185851Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46590
2022-11-23T02:16:44.4186105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46591
2022-11-23T02:16:44.4186510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4186724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4187135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4187343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4187747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4187957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4188415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4188645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4188927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4189209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4189656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4190092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4190338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4190602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4190896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8v9i3nuh
2022-11-23T02:16:44.4191202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8v9i3nuh/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4191491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qd5lhsi
2022-11-23T02:16:44.4191795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qd5lhsi/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4191938Z ok (5.600s)
2022-11-23T02:16:44.4191958Z 
2022-11-23T02:16:44.4192263Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4192391Z Ran 1 test in 5.601s
2022-11-23T02:16:44.4192430Z 
2022-11-23T02:16:44.4192544Z OK
2022-11-23T02:16:44.4192562Z 
2022-11-23T02:16:44.4192722Z Generating XML reports...
2022-11-23T02:16:44.4193204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021109.xml
2022-11-23T02:16:44.4193612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4193824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4194237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4194463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4194529Z 
2022-11-23T02:16:44.4194680Z Running tests...
2022-11-23T02:16:44.4194961Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4195311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4195617Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4195871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46705
2022-11-23T02:16:44.4196128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46706
2022-11-23T02:16:44.4196535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4196745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4197158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4197369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4197774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4197985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4198395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4198618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4198943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4199230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4199670Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4200107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4200354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4200615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4200754Z ok (5.661s)
2022-11-23T02:16:44.4200774Z 
2022-11-23T02:16:44.4201073Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4201218Z Ran 1 test in 5.662s
2022-11-23T02:16:44.4201241Z 
2022-11-23T02:16:44.4201370Z OK
2022-11-23T02:16:44.4201390Z 
2022-11-23T02:16:44.4201545Z Generating XML reports...
2022-11-23T02:16:44.4202029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021118.xml
2022-11-23T02:16:44.4202434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4202633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4203048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4203275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4203294Z 
2022-11-23T02:16:44.4203427Z Running tests...
2022-11-23T02:16:44.4203718Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4204064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4204387Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4204641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46817
2022-11-23T02:16:44.4204873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46818
2022-11-23T02:16:44.4205329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4205535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4206157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4206379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4206777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4206977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4207386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4207604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4207872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4208152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4208591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4209021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4209355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4209629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4209771Z ok (5.564s)
2022-11-23T02:16:44.4209791Z 
2022-11-23T02:16:44.4210092Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4210225Z Ran 1 test in 5.564s
2022-11-23T02:16:44.4210265Z 
2022-11-23T02:16:44.4210377Z OK
2022-11-23T02:16:44.4210395Z 
2022-11-23T02:16:44.4210553Z Generating XML reports...
2022-11-23T02:16:44.4211042Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021126.xml
2022-11-23T02:16:44.4211444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4211651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4212067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4212292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4212312Z 
2022-11-23T02:16:44.4212456Z Running tests...
2022-11-23T02:16:44.4212736Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4213087Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4213401Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4214181Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.616s)
2022-11-23T02:16:44.4214202Z 
2022-11-23T02:16:44.4214501Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4214650Z Ran 1 test in 1.616s
2022-11-23T02:16:44.4214669Z 
2022-11-23T02:16:44.4214803Z OK (skipped=1)
2022-11-23T02:16:44.4214822Z 
2022-11-23T02:16:44.4214980Z Generating XML reports...
2022-11-23T02:16:44.4215465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021134.xml
2022-11-23T02:16:44.4215938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4216129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4216544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4216812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4216832Z 
2022-11-23T02:16:44.4216977Z Running tests...
2022-11-23T02:16:44.4217277Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4217626Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4217956Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4218736Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.636s)
2022-11-23T02:16:44.4218760Z 
2022-11-23T02:16:44.4219052Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4219181Z Ran 1 test in 1.636s
2022-11-23T02:16:44.4219220Z 
2022-11-23T02:16:44.4219336Z OK (skipped=1)
2022-11-23T02:16:44.4219355Z 
2022-11-23T02:16:44.4219512Z Generating XML reports...
2022-11-23T02:16:44.4220047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021138.xml
2022-11-23T02:16:44.4220459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4220670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4221089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4221317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4221337Z 
2022-11-23T02:16:44.4221482Z Running tests...
2022-11-23T02:16:44.4221757Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4222106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4222458Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4222714Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46997
2022-11-23T02:16:44.4222965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46998
2022-11-23T02:16:44.4223372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4223586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4223998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4224204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4224607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4224816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4225234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4225459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4225737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4226068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4226507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4226934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4227183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4227449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4227642Z skip: Need at least 4 CUDA devices (4.092s)
2022-11-23T02:16:44.4227662Z 
2022-11-23T02:16:44.4227968Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4228117Z Ran 1 test in 4.092s
2022-11-23T02:16:44.4228136Z 
2022-11-23T02:16:44.4228280Z OK (skipped=1)
2022-11-23T02:16:44.4228303Z 
2022-11-23T02:16:44.4228463Z Generating XML reports...
2022-11-23T02:16:44.4228945Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021142.xml
2022-11-23T02:16:44.4229348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4229538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4229952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4230225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4230246Z 
2022-11-23T02:16:44.4230396Z Running tests...
2022-11-23T02:16:44.4230694Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4231044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4231410Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4231665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47106
2022-11-23T02:16:44.4231899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47107
2022-11-23T02:16:44.4232303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4232510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4232919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4233143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4233540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4233753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4234160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4234384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4234642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4234919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4235355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4235780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4236039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4236349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4236530Z skip: Need at least 4 CUDA devices (4.153s)
2022-11-23T02:16:44.4236549Z 
2022-11-23T02:16:44.4236853Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4236995Z Ran 1 test in 4.153s
2022-11-23T02:16:44.4237015Z 
2022-11-23T02:16:44.4237139Z OK (skipped=1)
2022-11-23T02:16:44.4237158Z 
2022-11-23T02:16:44.4237312Z Generating XML reports...
2022-11-23T02:16:44.4237794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021149.xml
2022-11-23T02:16:44.4238194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4238399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4238805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4239028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4239048Z 
2022-11-23T02:16:44.4239180Z Running tests...
2022-11-23T02:16:44.4239453Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4239797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4240112Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4240937Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.590s)
2022-11-23T02:16:44.4240959Z 
2022-11-23T02:16:44.4241252Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4241398Z Ran 1 test in 1.590s
2022-11-23T02:16:44.4241418Z 
2022-11-23T02:16:44.4241561Z OK (skipped=1)
2022-11-23T02:16:44.4241580Z 
2022-11-23T02:16:44.4241739Z Generating XML reports...
2022-11-23T02:16:44.4242217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021155.xml
2022-11-23T02:16:44.4242622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4242808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4243221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4243444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4243464Z 
2022-11-23T02:16:44.4243597Z Running tests...
2022-11-23T02:16:44.4243885Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4244232Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4244527Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4244774Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47249
2022-11-23T02:16:44.4245022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47250
2022-11-23T02:16:44.4245408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4245615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4246250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4246473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4246967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4247169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4247578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4247799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4248053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4248339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4248775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4249199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4249464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4249724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4249995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.4250264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.4250692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4251154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4251535Z STAGE:2022-11-23 02:12:04 47249:47249 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4251893Z STAGE:2022-11-23 02:12:04 47250:47250 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4252264Z STAGE:2022-11-23 02:12:04 47250:47250 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4252621Z STAGE:2022-11-23 02:12:04 47249:47249 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4253001Z STAGE:2022-11-23 02:12:04 47250:47250 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4253378Z STAGE:2022-11-23 02:12:04 47249:47249 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4253733Z STAGE:2022-11-23 02:12:04 47250:47250 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4254083Z STAGE:2022-11-23 02:12:04 47249:47249 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4254636Z STAGE:2022-11-23 02:12:04 47249:47249 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:12:04 47250:47250 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4254681Z 
2022-11-23T02:16:44.4255259Z STAGE:2022-11-23 02:12:04 47249:47249 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:12:04 47250:47250 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4255300Z 
2022-11-23T02:16:44.4255422Z ok (4.421s)
2022-11-23T02:16:44.4255441Z 
2022-11-23T02:16:44.4255735Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4255876Z Ran 1 test in 4.422s
2022-11-23T02:16:44.4255896Z 
2022-11-23T02:16:44.4256021Z OK
2022-11-23T02:16:44.4256040Z 
2022-11-23T02:16:44.4256195Z Generating XML reports...
2022-11-23T02:16:44.4256676Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021200.xml
2022-11-23T02:16:44.4257077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4257334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4257732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4257955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4257974Z 
2022-11-23T02:16:44.4258110Z Running tests...
2022-11-23T02:16:44.4258400Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4258744Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4259045Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4259297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47368
2022-11-23T02:16:44.4259545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47369
2022-11-23T02:16:44.4259950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4260144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4260557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4260778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4261180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4261387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4261842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4262068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4262352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4262613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4263054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4263487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4263748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4264012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4264289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.4264560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.4264988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4265417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4265782Z STAGE:2022-11-23 02:12:10 47368:47368 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4266113Z STAGE:2022-11-23 02:12:10 47369:47369 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4266483Z STAGE:2022-11-23 02:12:10 47369:47369 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4266841Z STAGE:2022-11-23 02:12:10 47368:47368 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4267218Z STAGE:2022-11-23 02:12:10 47369:47369 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4267594Z STAGE:2022-11-23 02:12:10 47368:47368 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4268006Z STAGE:2022-11-23 02:12:10 47369:47369 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4268355Z STAGE:2022-11-23 02:12:10 47368:47368 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4268716Z STAGE:2022-11-23 02:12:10 47368:47368 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4269052Z STAGE:2022-11-23 02:12:10 47369:47369 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4269425Z STAGE:2022-11-23 02:12:10 47368:47368 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4269800Z STAGE:2022-11-23 02:12:10 47369:47369 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4269940Z ok (4.242s)
2022-11-23T02:16:44.4269959Z 
2022-11-23T02:16:44.4270254Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4270398Z Ran 1 test in 4.243s
2022-11-23T02:16:44.4270420Z 
2022-11-23T02:16:44.4270543Z OK
2022-11-23T02:16:44.4270563Z 
2022-11-23T02:16:44.4270715Z Generating XML reports...
2022-11-23T02:16:44.4271198Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021207.xml
2022-11-23T02:16:44.4271582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4271793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4272249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4272477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4272497Z 
2022-11-23T02:16:44.4272641Z Running tests...
2022-11-23T02:16:44.4272937Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4273284Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4273593Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4273826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47487
2022-11-23T02:16:44.4274074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47488
2022-11-23T02:16:44.4274478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4274682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4275092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4275312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4275713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4275920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4276334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4276533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4276808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4277081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4277524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4277956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4278216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4278530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4278804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.4279077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.4279490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4279917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4280293Z STAGE:2022-11-23 02:12:17 47487:47487 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4280647Z STAGE:2022-11-23 02:12:17 47488:47488 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4281013Z STAGE:2022-11-23 02:12:17 47488:47488 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4281378Z STAGE:2022-11-23 02:12:17 47487:47487 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4281756Z STAGE:2022-11-23 02:12:17 47488:47488 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4282134Z STAGE:2022-11-23 02:12:17 47487:47487 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4282487Z STAGE:2022-11-23 02:12:17 47488:47488 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4282864Z STAGE:2022-11-23 02:12:17 47487:47487 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4283238Z STAGE:2022-11-23 02:12:17 47487:47487 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4283598Z STAGE:2022-11-23 02:12:17 47488:47488 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4283974Z STAGE:2022-11-23 02:12:17 47487:47487 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4284347Z STAGE:2022-11-23 02:12:17 47488:47488 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4284485Z ok (4.247s)
2022-11-23T02:16:44.4284505Z 
2022-11-23T02:16:44.4284798Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4284939Z Ran 1 test in 4.247s
2022-11-23T02:16:44.4284960Z 
2022-11-23T02:16:44.4285076Z OK
2022-11-23T02:16:44.4285095Z 
2022-11-23T02:16:44.4285231Z Generating XML reports...
2022-11-23T02:16:44.4285928Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021213.xml
2022-11-23T02:16:44.4286344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4286553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4286972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4287192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4287212Z 
2022-11-23T02:16:44.4287358Z Running tests...
2022-11-23T02:16:44.4287650Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4287976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4288273Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4288519Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47606
2022-11-23T02:16:44.4288768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47607
2022-11-23T02:16:44.4289174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4289459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4289873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4290094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4290490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4290676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4291092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4291310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4291588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4291864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4292308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4292732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4292998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4293256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4293567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.4293846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.4294278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4294706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4295077Z STAGE:2022-11-23 02:12:24 47606:47606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4295431Z STAGE:2022-11-23 02:12:24 47607:47607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4296010Z STAGE:2022-11-23 02:12:24 47606:47606 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:12:24 47607:47607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4296030Z 
2022-11-23T02:16:44.4296640Z STAGE:2022-11-23 02:12:24 47606:47606 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:12:24 47607:47607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4296661Z 
2022-11-23T02:16:44.4297017Z STAGE:2022-11-23 02:12:24 47607:47607 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4297373Z STAGE:2022-11-23 02:12:24 47606:47606 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4297720Z STAGE:2022-11-23 02:12:24 47607:47607 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4298084Z STAGE:2022-11-23 02:12:24 47606:47606 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4298457Z STAGE:2022-11-23 02:12:24 47607:47607 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4298834Z STAGE:2022-11-23 02:12:24 47606:47606 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4298973Z ok (4.251s)
2022-11-23T02:16:44.4298992Z 
2022-11-23T02:16:44.4299285Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4299430Z Ran 1 test in 4.252s
2022-11-23T02:16:44.4299450Z 
2022-11-23T02:16:44.4299565Z OK
2022-11-23T02:16:44.4299631Z 
2022-11-23T02:16:44.4299790Z Generating XML reports...
2022-11-23T02:16:44.4300256Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021220.xml
2022-11-23T02:16:44.4300665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4300872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4301286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4301507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4301526Z 
2022-11-23T02:16:44.4301662Z Running tests...
2022-11-23T02:16:44.4301953Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4302300Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4302578Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4302827Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47725
2022-11-23T02:16:44.4303072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47726
2022-11-23T02:16:44.4303476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4303676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4304130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4304356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4304758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4304965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4305360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4305580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4305855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4306125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4306557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4306990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4307252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4307514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4307718Z skip: Skipped due to small world size. (4.265s)
2022-11-23T02:16:44.4307738Z 
2022-11-23T02:16:44.4308015Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4308165Z Ran 1 test in 4.265s
2022-11-23T02:16:44.4308185Z 
2022-11-23T02:16:44.4308317Z OK (skipped=1)
2022-11-23T02:16:44.4308335Z 
2022-11-23T02:16:44.4308496Z Generating XML reports...
2022-11-23T02:16:44.4308977Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021227.xml
2022-11-23T02:16:44.4309384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4309590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4310005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4310260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4310302Z 
2022-11-23T02:16:44.4310428Z Running tests...
2022-11-23T02:16:44.4310723Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4311066Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4311360Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4311610Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47834
2022-11-23T02:16:44.4311859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47835
2022-11-23T02:16:44.4312265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4312467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4312859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4313085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4313484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4313687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4314096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4314363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4314648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4314918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4315334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4315770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4316028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4316281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4316466Z skip: Skipped due to small world size. (4.247s)
2022-11-23T02:16:44.4316486Z 
2022-11-23T02:16:44.4316828Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4316974Z Ran 1 test in 4.247s
2022-11-23T02:16:44.4316993Z 
2022-11-23T02:16:44.4317138Z OK (skipped=1)
2022-11-23T02:16:44.4317157Z 
2022-11-23T02:16:44.4317305Z Generating XML reports...
2022-11-23T02:16:44.4317768Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021234.xml
2022-11-23T02:16:44.4318175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4318380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4318790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4319016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4319035Z 
2022-11-23T02:16:44.4319179Z Running tests...
2022-11-23T02:16:44.4319462Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4319811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4320111Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4320345Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47943
2022-11-23T02:16:44.4320660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47944
2022-11-23T02:16:44.4321066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4321272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4321682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4321902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4322298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4322502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4322895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4323117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4323392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4323666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4324101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4324528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4324833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4325099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4325289Z skip: Skipped due to small world size. (4.250s)
2022-11-23T02:16:44.4325308Z 
2022-11-23T02:16:44.4325589Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4325942Z Ran 1 test in 4.250s
2022-11-23T02:16:44.4325965Z 
2022-11-23T02:16:44.4326110Z OK (skipped=1)
2022-11-23T02:16:44.4326129Z 
2022-11-23T02:16:44.4326278Z Generating XML reports...
2022-11-23T02:16:44.4326762Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021240.xml
2022-11-23T02:16:44.4327160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4327365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4327781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4328004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4328024Z 
2022-11-23T02:16:44.4328149Z Running tests...
2022-11-23T02:16:44.4328447Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4328789Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4329073Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4329321Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48052
2022-11-23T02:16:44.4329568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48053
2022-11-23T02:16:44.4329963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4330168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4330557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4330775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4331254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4331457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4331863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4332082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4332356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4332634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4333073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4333485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4333748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4334005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4334192Z skip: Skipped due to small world size. (4.246s)
2022-11-23T02:16:44.4334211Z 
2022-11-23T02:16:44.4334504Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4334644Z Ran 1 test in 4.246s
2022-11-23T02:16:44.4334664Z 
2022-11-23T02:16:44.4334795Z OK (skipped=1)
2022-11-23T02:16:44.4334814Z 
2022-11-23T02:16:44.4335027Z Generating XML reports...
2022-11-23T02:16:44.4335503Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021247.xml
2022-11-23T02:16:44.4335905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4336117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4336527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4336743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4336763Z 
2022-11-23T02:16:44.4336893Z Running tests...
2022-11-23T02:16:44.4337183Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4337526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4337810Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4338042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48161
2022-11-23T02:16:44.4338289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48162
2022-11-23T02:16:44.4338696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4338904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4339313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4339530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4339921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4340125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4340516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4340736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4341010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4341348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4341784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4342207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4342469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4342731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4343105Z STAGE:2022-11-23 02:12:58 48162:48162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4343439Z STAGE:2022-11-23 02:12:58 48161:48161 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4343806Z STAGE:2022-11-23 02:12:58 48162:48162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4344168Z STAGE:2022-11-23 02:12:58 48161:48161 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4344771Z STAGE:2022-11-23 02:12:58 48162:48162 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:12:58 48161:48161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4344792Z 
2022-11-23T02:16:44.4345144Z STAGE:2022-11-23 02:12:58 48162:48162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4345567Z STAGE:2022-11-23 02:12:58 48161:48161 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4345936Z STAGE:2022-11-23 02:12:58 48161:48161 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4346294Z STAGE:2022-11-23 02:12:58 48162:48162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4346666Z STAGE:2022-11-23 02:12:58 48161:48161 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4347055Z STAGE:2022-11-23 02:12:58 48162:48162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4347175Z ok (4.304s)
2022-11-23T02:16:44.4347194Z 
2022-11-23T02:16:44.4347491Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4347632Z Ran 1 test in 4.305s
2022-11-23T02:16:44.4347652Z 
2022-11-23T02:16:44.4347778Z OK
2022-11-23T02:16:44.4347796Z 
2022-11-23T02:16:44.4347948Z Generating XML reports...
2022-11-23T02:16:44.4348429Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021254.xml
2022-11-23T02:16:44.4348834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4349040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4349455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4349662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4349681Z 
2022-11-23T02:16:44.4349816Z Running tests...
2022-11-23T02:16:44.4350106Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4350448Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4350726Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4350982Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48274
2022-11-23T02:16:44.4351232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48275
2022-11-23T02:16:44.4351636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4351880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4352295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4352515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4352917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4353123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4353535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4353755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4354031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4354288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4354725Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4355154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4355413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4355780Z STAGE:2022-11-23 02:13:05 48275:48275 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4356085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4356460Z STAGE:2022-11-23 02:13:05 48274:48274 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4357043Z STAGE:2022-11-23 02:13:05 48274:48274 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:13:05 48275:48275 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4357068Z 
2022-11-23T02:16:44.4357664Z STAGE:2022-11-23 02:13:05 48274:48274 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:13:05 48275:48275 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4357683Z 
2022-11-23T02:16:44.4358040Z STAGE:2022-11-23 02:13:05 48275:48275 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4358396Z STAGE:2022-11-23 02:13:05 48274:48274 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4358743Z STAGE:2022-11-23 02:13:05 48275:48275 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4359097Z STAGE:2022-11-23 02:13:05 48274:48274 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4359470Z STAGE:2022-11-23 02:13:05 48275:48275 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4359850Z STAGE:2022-11-23 02:13:05 48274:48274 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4359989Z ok (4.311s)
2022-11-23T02:16:44.4360009Z 
2022-11-23T02:16:44.4360306Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4360446Z Ran 1 test in 4.311s
2022-11-23T02:16:44.4360465Z 
2022-11-23T02:16:44.4360582Z OK
2022-11-23T02:16:44.4360601Z 
2022-11-23T02:16:44.4360754Z Generating XML reports...
2022-11-23T02:16:44.4361223Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021301.xml
2022-11-23T02:16:44.4361632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4361838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4362251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4362526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4362546Z 
2022-11-23T02:16:44.4362690Z Running tests...
2022-11-23T02:16:44.4362990Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4363335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4363629Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-11-23T02:16:44.4363671Z 
2022-11-23T02:16:44.4363944Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4364082Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4364101Z 
2022-11-23T02:16:44.4364243Z OK (skipped=1)
2022-11-23T02:16:44.4364261Z 
2022-11-23T02:16:44.4364412Z Generating XML reports...
2022-11-23T02:16:44.4364890Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021308.xml
2022-11-23T02:16:44.4365273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4365475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4366099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4366319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4366339Z 
2022-11-23T02:16:44.4366481Z Running tests...
2022-11-23T02:16:44.4366774Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4367189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4367482Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4367731Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48420
2022-11-23T02:16:44.4367970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48421
2022-11-23T02:16:44.4368373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4368573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4368980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4369198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4369594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4369793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4370199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4370400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4370676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4370947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4371382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4371812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4372072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4372327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4372694Z STAGE:2022-11-23 02:13:14 48420:48420 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4373112Z STAGE:2022-11-23 02:13:14 48421:48421 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4373668Z STAGE:2022-11-23 02:13:14 48421:48421 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:13:14 48420:48420 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4373706Z 
2022-11-23T02:16:44.4374293Z STAGE:2022-11-23 02:13:14 48420:48420 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:13:14 48421:48421 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4374330Z 
2022-11-23T02:16:44.4374673Z STAGE:2022-11-23 02:13:14 48421:48421 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4375024Z STAGE:2022-11-23 02:13:14 48420:48420 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4375384Z STAGE:2022-11-23 02:13:14 48421:48421 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4375741Z STAGE:2022-11-23 02:13:14 48420:48420 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4376112Z STAGE:2022-11-23 02:13:14 48421:48421 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4376485Z STAGE:2022-11-23 02:13:14 48420:48420 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4376618Z ok (4.261s)
2022-11-23T02:16:44.4376637Z 
2022-11-23T02:16:44.4376925Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4377069Z Ran 1 test in 4.262s
2022-11-23T02:16:44.4377089Z 
2022-11-23T02:16:44.4377233Z OK
2022-11-23T02:16:44.4377254Z 
2022-11-23T02:16:44.4377408Z Generating XML reports...
2022-11-23T02:16:44.4377891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021310.xml
2022-11-23T02:16:44.4378290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4378498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4378911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4379130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4379150Z 
2022-11-23T02:16:44.4379291Z Running tests...
2022-11-23T02:16:44.4379567Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4379909Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4380238Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2022-11-23T02:16:44.4380258Z 
2022-11-23T02:16:44.4380544Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4380689Z Ran 1 test in 0.003s
2022-11-23T02:16:44.4380712Z 
2022-11-23T02:16:44.4380852Z OK (skipped=1)
2022-11-23T02:16:44.4380871Z 
2022-11-23T02:16:44.4381018Z Generating XML reports...
2022-11-23T02:16:44.4381501Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021317.xml
2022-11-23T02:16:44.4381902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4382087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4382501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4382718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4382737Z 
2022-11-23T02:16:44.4382865Z Running tests...
2022-11-23T02:16:44.4383147Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4383545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4383847Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-11-23T02:16:44.4383867Z 
2022-11-23T02:16:44.4384151Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4384288Z Ran 1 test in 0.003s
2022-11-23T02:16:44.4384307Z 
2022-11-23T02:16:44.4384431Z OK (skipped=1)
2022-11-23T02:16:44.4384449Z 
2022-11-23T02:16:44.4384594Z Generating XML reports...
2022-11-23T02:16:44.4385072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021319.xml
2022-11-23T02:16:44.4385469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4385666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4386076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4386298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4386318Z 
2022-11-23T02:16:44.4386457Z Running tests...
2022-11-23T02:16:44.4386730Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4387070Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4387345Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4387638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48599
2022-11-23T02:16:44.4387891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48600
2022-11-23T02:16:44.4388293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4388494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4388905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4389122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4389503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4389707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4390118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4390332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4390607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4390878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4391312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4391736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4391993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4392236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4392602Z STAGE:2022-11-23 02:13:26 48599:48599 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4392957Z STAGE:2022-11-23 02:13:26 48600:48600 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4393528Z STAGE:2022-11-23 02:13:26 48599:48599 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:13:26 48600:48600 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4393608Z 
2022-11-23T02:16:44.4394217Z STAGE:2022-11-23 02:13:26 48600:48600 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:13:26 48599:48599 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4394237Z 
2022-11-23T02:16:44.4394592Z STAGE:2022-11-23 02:13:26 48600:48600 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4394937Z STAGE:2022-11-23 02:13:26 48599:48599 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4395300Z STAGE:2022-11-23 02:13:26 48599:48599 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4395649Z STAGE:2022-11-23 02:13:26 48600:48600 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4396023Z STAGE:2022-11-23 02:13:26 48599:48599 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4396385Z STAGE:2022-11-23 02:13:26 48600:48600 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4396510Z ok (4.261s)
2022-11-23T02:16:44.4396530Z 
2022-11-23T02:16:44.4396819Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4396957Z Ran 1 test in 4.261s
2022-11-23T02:16:44.4396976Z 
2022-11-23T02:16:44.4397096Z OK
2022-11-23T02:16:44.4397114Z 
2022-11-23T02:16:44.4397262Z Generating XML reports...
2022-11-23T02:16:44.4397740Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021322.xml
2022-11-23T02:16:44.4398184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4398394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4398791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4399015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4399035Z 
2022-11-23T02:16:44.4399165Z Running tests...
2022-11-23T02:16:44.4399451Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4399789Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4400078Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-11-23T02:16:44.4400098Z 
2022-11-23T02:16:44.4400380Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4400519Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4400538Z 
2022-11-23T02:16:44.4400654Z OK (skipped=1)
2022-11-23T02:16:44.4400688Z 
2022-11-23T02:16:44.4400818Z Generating XML reports...
2022-11-23T02:16:44.4401292Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021329.xml
2022-11-23T02:16:44.4401692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4401888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4402290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4402503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4402522Z 
2022-11-23T02:16:44.4402651Z Running tests...
2022-11-23T02:16:44.4402938Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4403265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4403615Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-11-23T02:16:44.4403635Z 
2022-11-23T02:16:44.4403926Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4404113Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4404132Z 
2022-11-23T02:16:44.4404269Z OK (skipped=1)
2022-11-23T02:16:44.4404287Z 
2022-11-23T02:16:44.4404439Z Generating XML reports...
2022-11-23T02:16:44.4404922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021331.xml
2022-11-23T02:16:44.4405317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4405521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4406128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4406352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4406373Z 
2022-11-23T02:16:44.4406515Z Running tests...
2022-11-23T02:16:44.4406805Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4407150Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4407435Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4407683Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48778
2022-11-23T02:16:44.4407928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48779
2022-11-23T02:16:44.4408309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4408584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4409002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4409217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4409616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4409815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4410224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4410441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4410714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4410977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4411410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4411833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4412093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4412454Z STAGE:2022-11-23 02:13:37 48779:48779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4412709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4413070Z STAGE:2022-11-23 02:13:37 48778:48778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4413650Z STAGE:2022-11-23 02:13:37 48778:48778 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:13:37 48779:48779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4413670Z 
2022-11-23T02:16:44.4414272Z STAGE:2022-11-23 02:13:37 48779:48779 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:13:37 48778:48778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4414357Z 
2022-11-23T02:16:44.4414719Z STAGE:2022-11-23 02:13:37 48779:48779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4415047Z STAGE:2022-11-23 02:13:37 48778:48778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4415410Z STAGE:2022-11-23 02:13:37 48778:48778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4415984Z STAGE:2022-11-23 02:13:37 48779:48779 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:13:37 48778:48778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4416004Z 
2022-11-23T02:16:44.4416380Z STAGE:2022-11-23 02:13:37 48779:48779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4416516Z ok (4.266s)
2022-11-23T02:16:44.4416535Z 
2022-11-23T02:16:44.4416870Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4417008Z Ran 1 test in 4.266s
2022-11-23T02:16:44.4417031Z 
2022-11-23T02:16:44.4417148Z OK
2022-11-23T02:16:44.4417167Z 
2022-11-23T02:16:44.4417318Z Generating XML reports...
2022-11-23T02:16:44.4417798Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021333.xml
2022-11-23T02:16:44.4418182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4418385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4418793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4419059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4419080Z 
2022-11-23T02:16:44.4419224Z Running tests...
2022-11-23T02:16:44.4419516Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4419855Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4420127Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4420359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48891
2022-11-23T02:16:44.4420603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48892
2022-11-23T02:16:44.4421005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4421205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4421613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4421827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4422212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4422413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4422821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4423024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4423299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4423569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4424000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4424430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4424692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4425002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4425368Z STAGE:2022-11-23 02:13:44 48892:48892 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4425716Z STAGE:2022-11-23 02:13:44 48891:48891 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4426065Z STAGE:2022-11-23 02:13:44 48891:48891 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4426423Z STAGE:2022-11-23 02:13:44 48892:48892 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4426797Z STAGE:2022-11-23 02:13:44 48891:48891 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4427174Z STAGE:2022-11-23 02:13:44 48892:48892 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4427728Z STAGE:2022-11-23 02:13:44 48892:48892 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 02:13:44 48891:48891 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4427751Z 
2022-11-23T02:16:44.4428105Z STAGE:2022-11-23 02:13:44 48892:48892 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4428459Z STAGE:2022-11-23 02:13:44 48891:48891 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4428828Z STAGE:2022-11-23 02:13:44 48892:48892 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4429200Z STAGE:2022-11-23 02:13:44 48891:48891 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4429365Z ok (4.202s)
2022-11-23T02:16:44.4429403Z 
2022-11-23T02:16:44.4429686Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4429825Z Ran 1 test in 4.202s
2022-11-23T02:16:44.4429844Z 
2022-11-23T02:16:44.4429968Z OK
2022-11-23T02:16:44.4429987Z 
2022-11-23T02:16:44.4430140Z Generating XML reports...
2022-11-23T02:16:44.4430624Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021340.xml
2022-11-23T02:16:44.4431020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4431223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4431632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4431836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4431859Z 
2022-11-23T02:16:44.4431991Z Running tests...
2022-11-23T02:16:44.4432280Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4432624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4432907Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4433158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49004
2022-11-23T02:16:44.4433401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49005
2022-11-23T02:16:44.4433803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4433989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4434387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4434594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4435000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4435215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4435672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4435886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4436160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4436427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4436843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4437269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4437533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4437788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4437924Z ok (4.167s)
2022-11-23T02:16:44.4437944Z 
2022-11-23T02:16:44.4438235Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4438375Z Ran 1 test in 4.167s
2022-11-23T02:16:44.4438394Z 
2022-11-23T02:16:44.4438515Z OK
2022-11-23T02:16:44.4438534Z 
2022-11-23T02:16:44.4438669Z Generating XML reports...
2022-11-23T02:16:44.4439151Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021347.xml
2022-11-23T02:16:44.4439595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4439801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4440205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4440417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4440441Z 
2022-11-23T02:16:44.4440582Z Running tests...
2022-11-23T02:16:44.4440869Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4441209Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4441482Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4441730Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49113
2022-11-23T02:16:44.4441972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49114
2022-11-23T02:16:44.4442372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4442572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4442975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4443196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4443593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4443779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4444189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4444407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4444684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4444957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4445388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4446080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4446342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4446598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4446948Z STAGE:2022-11-23 02:13:58 49113:49113 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4447293Z STAGE:2022-11-23 02:13:58 49114:49114 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4447866Z STAGE:2022-11-23 02:13:58 49114:49114 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:13:58 49113:49113 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4447888Z 
2022-11-23T02:16:44.4448481Z STAGE:2022-11-23 02:13:58 49113:49113 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:13:58 49114:49114 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4448505Z 
2022-11-23T02:16:44.4448855Z STAGE:2022-11-23 02:13:58 49114:49114 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4449201Z STAGE:2022-11-23 02:13:58 49113:49113 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4449562Z STAGE:2022-11-23 02:13:58 49114:49114 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4449909Z STAGE:2022-11-23 02:13:58 49113:49113 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4450352Z STAGE:2022-11-23 02:13:58 49114:49114 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4450735Z STAGE:2022-11-23 02:13:58 49113:49113 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4450871Z ok (4.283s)
2022-11-23T02:16:44.4450892Z 
2022-11-23T02:16:44.4451175Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4451313Z Ran 1 test in 4.283s
2022-11-23T02:16:44.4451332Z 
2022-11-23T02:16:44.4451445Z OK
2022-11-23T02:16:44.4451464Z 
2022-11-23T02:16:44.4451610Z Generating XML reports...
2022-11-23T02:16:44.4452089Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021354.xml
2022-11-23T02:16:44.4452491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4452693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4453112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4453315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4453347Z 
2022-11-23T02:16:44.4453474Z Running tests...
2022-11-23T02:16:44.4453768Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4454103Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4454385Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T02:16:44.4454405Z 
2022-11-23T02:16:44.4454692Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4454835Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4454854Z 
2022-11-23T02:16:44.4454994Z OK (skipped=1)
2022-11-23T02:16:44.4455012Z 
2022-11-23T02:16:44.4455157Z Generating XML reports...
2022-11-23T02:16:44.4455628Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021401.xml
2022-11-23T02:16:44.4456029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4456232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4456715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4456930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4456949Z 
2022-11-23T02:16:44.4457077Z Running tests...
2022-11-23T02:16:44.4457363Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4457702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4457989Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T02:16:44.4458025Z 
2022-11-23T02:16:44.4458300Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4458446Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4458465Z 
2022-11-23T02:16:44.4458606Z OK (skipped=1)
2022-11-23T02:16:44.4458629Z 
2022-11-23T02:16:44.4458774Z Generating XML reports...
2022-11-23T02:16:44.4459251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021403.xml
2022-11-23T02:16:44.4459651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4459853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4460260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4460508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4460545Z 
2022-11-23T02:16:44.4460675Z Running tests...
2022-11-23T02:16:44.4460958Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4461297Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4461587Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4461832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49292
2022-11-23T02:16:44.4462074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49293
2022-11-23T02:16:44.4462474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4462673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4463066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4463281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4463667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4463871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4464281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4464494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4464770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4465044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4465457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4465887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4466152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4466404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4466744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.4467014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.4467436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4467853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4468222Z STAGE:2022-11-23 02:14:09 49292:49292 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4468573Z STAGE:2022-11-23 02:14:09 49293:49293 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4469130Z STAGE:2022-11-23 02:14:09 49292:49292 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:14:09 49293:49293 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4469168Z 
2022-11-23T02:16:44.4469755Z STAGE:2022-11-23 02:14:09 49292:49292 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:14:09 49293:49293 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4469775Z 
2022-11-23T02:16:44.4470131Z STAGE:2022-11-23 02:14:09 49292:49292 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4470472Z STAGE:2022-11-23 02:14:09 49293:49293 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4470876Z STAGE:2022-11-23 02:14:09 49293:49293 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4471238Z STAGE:2022-11-23 02:14:09 49292:49292 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4471613Z STAGE:2022-11-23 02:14:09 49293:49293 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4471990Z STAGE:2022-11-23 02:14:09 49292:49292 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4472125Z ok (4.264s)
2022-11-23T02:16:44.4472145Z 
2022-11-23T02:16:44.4472435Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4472559Z Ran 1 test in 4.265s
2022-11-23T02:16:44.4472577Z 
2022-11-23T02:16:44.4472702Z OK
2022-11-23T02:16:44.4472721Z 
2022-11-23T02:16:44.4472869Z Generating XML reports...
2022-11-23T02:16:44.4473350Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021405.xml
2022-11-23T02:16:44.4473749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4473954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4474367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4474589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4474609Z 
2022-11-23T02:16:44.4474739Z Running tests...
2022-11-23T02:16:44.4475012Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4475354Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4475637Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4475881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49411
2022-11-23T02:16:44.4476126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49412
2022-11-23T02:16:44.4476526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4476727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4477190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4477392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4477785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4477986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4478390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4478608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4478880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4479152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4479584Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4480015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4480259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4480512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4480691Z skip: Skipped due to small world size. (4.141s)
2022-11-23T02:16:44.4480710Z 
2022-11-23T02:16:44.4481038Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4481183Z Ran 1 test in 4.142s
2022-11-23T02:16:44.4481203Z 
2022-11-23T02:16:44.4481341Z OK (skipped=1)
2022-11-23T02:16:44.4481360Z 
2022-11-23T02:16:44.4481507Z Generating XML reports...
2022-11-23T02:16:44.4481985Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021412.xml
2022-11-23T02:16:44.4482375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4482579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4482988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4483204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4483223Z 
2022-11-23T02:16:44.4483351Z Running tests...
2022-11-23T02:16:44.4483637Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4483978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4484270Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4484519Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49520
2022-11-23T02:16:44.4484756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49521
2022-11-23T02:16:44.4485160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4485360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4485974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4486199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4486605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4486807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4487215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4487491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4487763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4488033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4488466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4488897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4489157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4489412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4489547Z ok (4.218s)
2022-11-23T02:16:44.4489567Z 
2022-11-23T02:16:44.4489865Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4489987Z Ran 1 test in 4.218s
2022-11-23T02:16:44.4490007Z 
2022-11-23T02:16:44.4490135Z OK
2022-11-23T02:16:44.4490154Z 
2022-11-23T02:16:44.4490304Z Generating XML reports...
2022-11-23T02:16:44.4490783Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021419.xml
2022-11-23T02:16:44.4491178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4491378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4491852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4492076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4492097Z 
2022-11-23T02:16:44.4492224Z Running tests...
2022-11-23T02:16:44.4492504Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4492847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4493116Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4493357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49629
2022-11-23T02:16:44.4493598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49630
2022-11-23T02:16:44.4493994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4494199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4494603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4494806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4495209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4495410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4495818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4496032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4496300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4496578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4497012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4497441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4497741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4497994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4498130Z ok (4.256s)
2022-11-23T02:16:44.4498149Z 
2022-11-23T02:16:44.4498442Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4498578Z Ran 1 test in 4.256s
2022-11-23T02:16:44.4498597Z 
2022-11-23T02:16:44.4498710Z OK
2022-11-23T02:16:44.4498729Z 
2022-11-23T02:16:44.4498875Z Generating XML reports...
2022-11-23T02:16:44.4499359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021426.xml
2022-11-23T02:16:44.4499741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4499945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4500351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4500570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4500590Z 
2022-11-23T02:16:44.4500725Z Running tests...
2022-11-23T02:16:44.4501012Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4501357Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4501649Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4501941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49738
2022-11-23T02:16:44.4502177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49739
2022-11-23T02:16:44.4502583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4502781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4503188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4503403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4503800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4504001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4504410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4504611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4504883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4505157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4505593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4506015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4506269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4506528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4506662Z ok (4.312s)
2022-11-23T02:16:44.4506686Z 
2022-11-23T02:16:44.4506975Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4507104Z Ran 1 test in 4.312s
2022-11-23T02:16:44.4507123Z 
2022-11-23T02:16:44.4507238Z OK
2022-11-23T02:16:44.4507258Z 
2022-11-23T02:16:44.4507413Z Generating XML reports...
2022-11-23T02:16:44.4507989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021432.xml
2022-11-23T02:16:44.4508382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4508581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4508983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4509194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4509214Z 
2022-11-23T02:16:44.4509347Z Running tests...
2022-11-23T02:16:44.4509620Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4509958Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4510278Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4510526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49847
2022-11-23T02:16:44.4510772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49848
2022-11-23T02:16:44.4511173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4511373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4511778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4512025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4512430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4512630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4513042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4513261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4513532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4513803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4514235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4514663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4514906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4515271Z STAGE:2022-11-23 02:14:43 49848:49848 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4515528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4515887Z STAGE:2022-11-23 02:14:43 49847:49847 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4516244Z STAGE:2022-11-23 02:14:43 49847:49847 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4516617Z STAGE:2022-11-23 02:14:43 49847:49847 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4517019Z STAGE:2022-11-23 02:14:43 49848:49848 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4517394Z STAGE:2022-11-23 02:14:43 49848:49848 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4517515Z ok (4.357s)
2022-11-23T02:16:44.4517548Z 
2022-11-23T02:16:44.4517821Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4517965Z Ran 1 test in 4.357s
2022-11-23T02:16:44.4518031Z 
2022-11-23T02:16:44.4518157Z OK
2022-11-23T02:16:44.4518176Z 
2022-11-23T02:16:44.4518323Z Generating XML reports...
2022-11-23T02:16:44.4518808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021439.xml
2022-11-23T02:16:44.4519208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4519411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4519822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4520027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4520059Z 
2022-11-23T02:16:44.4520186Z Running tests...
2022-11-23T02:16:44.4520474Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4520815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4521130Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4521375Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49960
2022-11-23T02:16:44.4521619Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49961
2022-11-23T02:16:44.4522015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4522216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4522659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4522882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4523270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4523469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4523879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4524092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4524364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4524633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4525053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4525481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4525945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4526324Z STAGE:2022-11-23 02:14:50 49961:49961 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4526582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4526940Z STAGE:2022-11-23 02:14:50 49960:49960 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4527305Z STAGE:2022-11-23 02:14:50 49960:49960 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4527894Z STAGE:2022-11-23 02:14:50 49961:49961 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:14:50 49960:49960 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4527915Z 
2022-11-23T02:16:44.4528291Z STAGE:2022-11-23 02:14:50 49961:49961 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4528426Z ok (4.264s)
2022-11-23T02:16:44.4528446Z 
2022-11-23T02:16:44.4528800Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4528938Z Ran 1 test in 4.264s
2022-11-23T02:16:44.4528958Z 
2022-11-23T02:16:44.4529083Z OK
2022-11-23T02:16:44.4529102Z 
2022-11-23T02:16:44.4529250Z Generating XML reports...
2022-11-23T02:16:44.4529731Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021446.xml
2022-11-23T02:16:44.4530130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4530331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4530743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4530947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4530982Z 
2022-11-23T02:16:44.4531097Z Running tests...
2022-11-23T02:16:44.4531392Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4531735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4532038Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4532287Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50073
2022-11-23T02:16:44.4532528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50074
2022-11-23T02:16:44.4532984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4533191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4533589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4533806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4534206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4534407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4534812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4535029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4535300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4535577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4535994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4536425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4536686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4537052Z STAGE:2022-11-23 02:14:57 50074:50074 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4537303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4537663Z STAGE:2022-11-23 02:14:57 50073:50073 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4538026Z STAGE:2022-11-23 02:14:57 50074:50074 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4538406Z STAGE:2022-11-23 02:14:57 50074:50074 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4538768Z STAGE:2022-11-23 02:14:57 50073:50073 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4539128Z STAGE:2022-11-23 02:14:57 50073:50073 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4539328Z ok (4.389s)
2022-11-23T02:16:44.4539347Z 
2022-11-23T02:16:44.4539638Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4539774Z Ran 1 test in 4.390s
2022-11-23T02:16:44.4539794Z 
2022-11-23T02:16:44.4539914Z OK
2022-11-23T02:16:44.4539933Z 
2022-11-23T02:16:44.4540077Z Generating XML reports...
2022-11-23T02:16:44.4540558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021453.xml
2022-11-23T02:16:44.4540960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4541165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4541558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4541774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4541798Z 
2022-11-23T02:16:44.4541938Z Running tests...
2022-11-23T02:16:44.4542227Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4542566Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4542828Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T02:16:44.4542848Z 
2022-11-23T02:16:44.4543136Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4543272Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4543291Z 
2022-11-23T02:16:44.4543452Z OK (skipped=1)
2022-11-23T02:16:44.4543488Z 
2022-11-23T02:16:44.4543627Z Generating XML reports...
2022-11-23T02:16:44.4544107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021500.xml
2022-11-23T02:16:44.4544502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4544709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4545115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4545330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4545350Z 
2022-11-23T02:16:44.4545480Z Running tests...
2022-11-23T02:16:44.4545762Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4546089Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4546386Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T02:16:44.4546406Z 
2022-11-23T02:16:44.4546692Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4546830Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4546849Z 
2022-11-23T02:16:44.4546990Z OK (skipped=1)
2022-11-23T02:16:44.4547009Z 
2022-11-23T02:16:44.4547164Z Generating XML reports...
2022-11-23T02:16:44.4547641Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021502.xml
2022-11-23T02:16:44.4548040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4548241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4548639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4548857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4548876Z 
2022-11-23T02:16:44.4549016Z Running tests...
2022-11-23T02:16:44.4549300Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4549697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4549984Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T02:16:44.4550004Z 
2022-11-23T02:16:44.4550286Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4550423Z Ran 1 test in 0.002s
2022-11-23T02:16:44.4550442Z 
2022-11-23T02:16:44.4550583Z OK (skipped=1)
2022-11-23T02:16:44.4550602Z 
2022-11-23T02:16:44.4550733Z Generating XML reports...
2022-11-23T02:16:44.4551217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021505.xml
2022-11-23T02:16:44.4551618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4551819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4552225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4552444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4552464Z 
2022-11-23T02:16:44.4552604Z Running tests...
2022-11-23T02:16:44.4552887Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4553212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4553512Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4553806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50285
2022-11-23T02:16:44.4554060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50286
2022-11-23T02:16:44.4554458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4554659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4555062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4555274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4555673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4555857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4556269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4556485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4556752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4557022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4557459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4557881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4558136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4558501Z STAGE:2022-11-23 02:15:11 50286:50286 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4558746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4559108Z STAGE:2022-11-23 02:15:11 50285:50285 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4559471Z STAGE:2022-11-23 02:15:11 50286:50286 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4568125Z STAGE:2022-11-23 02:15:11 50286:50286 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4568651Z STAGE:2022-11-23 02:15:11 50285:50285 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4569035Z STAGE:2022-11-23 02:15:11 50285:50285 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4569175Z ok (4.261s)
2022-11-23T02:16:44.4569197Z 
2022-11-23T02:16:44.4569495Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4569642Z Ran 1 test in 4.261s
2022-11-23T02:16:44.4569662Z 
2022-11-23T02:16:44.4569773Z OK
2022-11-23T02:16:44.4569792Z 
2022-11-23T02:16:44.4569953Z Generating XML reports...
2022-11-23T02:16:44.4570427Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021507.xml
2022-11-23T02:16:44.4570831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4571037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4571452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4571679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4571698Z 
2022-11-23T02:16:44.4571840Z Running tests...
2022-11-23T02:16:44.4572130Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4572482Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4572818Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4573085Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50398
2022-11-23T02:16:44.4573337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50399
2022-11-23T02:16:44.4573739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4573933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4574338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4574553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4574951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4575162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4575557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4575776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4576058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4576335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4576767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4577196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4577458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4577718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4577845Z ok (4.263s)
2022-11-23T02:16:44.4577865Z 
2022-11-23T02:16:44.4578147Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4578291Z Ran 1 test in 4.263s
2022-11-23T02:16:44.4578310Z 
2022-11-23T02:16:44.4578436Z OK
2022-11-23T02:16:44.4578505Z 
2022-11-23T02:16:44.4578653Z Generating XML reports...
2022-11-23T02:16:44.4579140Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021514.xml
2022-11-23T02:16:44.4579533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4579734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4580143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4580347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4580382Z 
2022-11-23T02:16:44.4580508Z Running tests...
2022-11-23T02:16:44.4580794Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4581140Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4581460Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4581709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50507
2022-11-23T02:16:44.4581958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50508
2022-11-23T02:16:44.4582365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4582565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4583008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4583234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4583634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4583832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4584245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4584460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4584730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4584998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4585422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4585853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4586116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4586481Z STAGE:2022-11-23 02:15:25 50508:50508 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4586746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4587112Z STAGE:2022-11-23 02:15:25 50507:50507 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4587472Z STAGE:2022-11-23 02:15:25 50507:50507 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4587848Z STAGE:2022-11-23 02:15:25 50507:50507 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4588216Z STAGE:2022-11-23 02:15:25 50508:50508 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4588577Z STAGE:2022-11-23 02:15:25 50508:50508 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4588698Z ok (4.457s)
2022-11-23T02:16:44.4588718Z 
2022-11-23T02:16:44.4589013Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4589199Z Ran 1 test in 4.457s
2022-11-23T02:16:44.4589218Z 
2022-11-23T02:16:44.4589341Z OK
2022-11-23T02:16:44.4589360Z 
2022-11-23T02:16:44.4589509Z Generating XML reports...
2022-11-23T02:16:44.4589996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021521.xml
2022-11-23T02:16:44.4590399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4590608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4591007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4591234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4591253Z 
2022-11-23T02:16:44.4591386Z Running tests...
2022-11-23T02:16:44.4591683Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4592031Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4592338Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4592586Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50620
2022-11-23T02:16:44.4592833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50621
2022-11-23T02:16:44.4593222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4593479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4593896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4594117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4594519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4594726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4595136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4595359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4595636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4595903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4596342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4596775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4597043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4597410Z STAGE:2022-11-23 02:15:32 50621:50621 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4597671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4598033Z STAGE:2022-11-23 02:15:32 50620:50620 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:16:44.4598397Z STAGE:2022-11-23 02:15:32 50620:50620 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4598778Z STAGE:2022-11-23 02:15:32 50620:50620 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4599131Z STAGE:2022-11-23 02:15:32 50621:50621 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:16:44.4599507Z STAGE:2022-11-23 02:15:32 50621:50621 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:16:44.4599693Z ok (4.350s)
2022-11-23T02:16:44.4599713Z 
2022-11-23T02:16:44.4600000Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4600147Z Ran 1 test in 4.350s
2022-11-23T02:16:44.4600167Z 
2022-11-23T02:16:44.4600294Z OK
2022-11-23T02:16:44.4600313Z 
2022-11-23T02:16:44.4600464Z Generating XML reports...
2022-11-23T02:16:44.4600954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021528.xml
2022-11-23T02:16:44.4601341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4601557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4601970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4602189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4602213Z 
2022-11-23T02:16:44.4602353Z Running tests...
2022-11-23T02:16:44.4602646Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4602996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4603293Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4603543Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50733
2022-11-23T02:16:44.4603778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50734
2022-11-23T02:16:44.4604226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4604445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4604858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4605085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4605482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4605935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4606366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4606573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4606859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4607138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4607576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4608009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4608269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4608527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4608663Z ok (4.200s)
2022-11-23T02:16:44.4608683Z 
2022-11-23T02:16:44.4608983Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4609113Z Ran 1 test in 4.200s
2022-11-23T02:16:44.4609133Z 
2022-11-23T02:16:44.4609261Z OK
2022-11-23T02:16:44.4609280Z 
2022-11-23T02:16:44.4609428Z Generating XML reports...
2022-11-23T02:16:44.4609918Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021535.xml
2022-11-23T02:16:44.4610319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4610607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4611023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4611243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4611262Z 
2022-11-23T02:16:44.4611404Z Running tests...
2022-11-23T02:16:44.4611685Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4612030Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4612336Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4612585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50932
2022-11-23T02:16:44.4612830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50933
2022-11-23T02:16:44.4613234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4613437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4613837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4614029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4614443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4614723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4615143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4615364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4615638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4615918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4616346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4616804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4617051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4617311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4617436Z ok (5.064s)
2022-11-23T02:16:44.4617456Z 
2022-11-23T02:16:44.4617743Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4617869Z Ran 1 test in 5.065s
2022-11-23T02:16:44.4617888Z 
2022-11-23T02:16:44.4618010Z OK
2022-11-23T02:16:44.4618029Z 
2022-11-23T02:16:44.4618180Z Generating XML reports...
2022-11-23T02:16:44.4618665Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021542.xml
2022-11-23T02:16:44.4619052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4619262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4619674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4619902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4619922Z 
2022-11-23T02:16:44.4620055Z Running tests...
2022-11-23T02:16:44.4620349Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4620695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4621067Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4621319Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51133
2022-11-23T02:16:44.4621555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51134
2022-11-23T02:16:44.4621961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4622163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4622578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4622803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4623206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4623411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4623825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4624032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4624310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4624584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4625060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4625497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4625753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4626017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4626150Z ok (5.608s)
2022-11-23T02:16:44.4626170Z 
2022-11-23T02:16:44.4626465Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4626596Z Ran 1 test in 5.608s
2022-11-23T02:16:44.4626615Z 
2022-11-23T02:16:44.4626741Z OK
2022-11-23T02:16:44.4626761Z 
2022-11-23T02:16:44.4626916Z Generating XML reports...
2022-11-23T02:16:44.4627402Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021549.xml
2022-11-23T02:16:44.4627805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4628012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4628422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4628651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4628671Z 
2022-11-23T02:16:44.4628815Z Running tests...
2022-11-23T02:16:44.4629094Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4629440Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4629734Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4629988Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51248
2022-11-23T02:16:44.4630240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51249
2022-11-23T02:16:44.4630648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4630857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4631267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4631530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4631924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4632131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4632539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4632760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4633035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4633310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4633747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4634178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4634425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4634715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatn71wse
2022-11-23T02:16:44.4635016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatn71wse/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4635318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4635610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivedafph
2022-11-23T02:16:44.4635917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivedafph/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4636052Z ok (4.369s)
2022-11-23T02:16:44.4636075Z 
2022-11-23T02:16:44.4636380Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4636507Z Ran 1 test in 4.369s
2022-11-23T02:16:44.4636542Z 
2022-11-23T02:16:44.4636652Z OK
2022-11-23T02:16:44.4636671Z 
2022-11-23T02:16:44.4636828Z Generating XML reports...
2022-11-23T02:16:44.4637313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021557.xml
2022-11-23T02:16:44.4637717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4637930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4638346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4638569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4638589Z 
2022-11-23T02:16:44.4638715Z Running tests...
2022-11-23T02:16:44.4638999Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4639345Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4639633Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4639881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51361
2022-11-23T02:16:44.4640131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51362
2022-11-23T02:16:44.4640539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4640747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4641156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4641362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4641815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4642020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4642426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4642650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4642929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4643207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4643642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4644069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4644320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4644582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4644873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpogwig2oy
2022-11-23T02:16:44.4645176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpogwig2oy/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4645464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24c6w9sd
2022-11-23T02:16:44.4646066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24c6w9sd/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4646212Z ok (5.019s)
2022-11-23T02:16:44.4646232Z 
2022-11-23T02:16:44.4646542Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4646690Z Ran 1 test in 5.020s
2022-11-23T02:16:44.4646714Z 
2022-11-23T02:16:44.4646824Z OK
2022-11-23T02:16:44.4646843Z 
2022-11-23T02:16:44.4646998Z Generating XML reports...
2022-11-23T02:16:44.4647486Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021604.xml
2022-11-23T02:16:44.4647889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4648095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4648506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4648726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4648745Z 
2022-11-23T02:16:44.4648886Z Running tests...
2022-11-23T02:16:44.4649168Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4649514Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4649848Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4650103Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51472
2022-11-23T02:16:44.4650355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51473
2022-11-23T02:16:44.4650758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4650961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4651375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4651597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4651981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4652262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4652673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4652897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4653176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4653453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4653893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4654324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4654582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4654831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4655124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz6947sts
2022-11-23T02:16:44.4655424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz6947sts/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4655711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_sov2vt
2022-11-23T02:16:44.4656009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_sov2vt/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4656871Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:16:44.4657700Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:16:44.4657837Z ok (5.534s)
2022-11-23T02:16:44.4657858Z 
2022-11-23T02:16:44.4658160Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4658306Z Ran 1 test in 5.534s
2022-11-23T02:16:44.4658330Z 
2022-11-23T02:16:44.4658455Z OK
2022-11-23T02:16:44.4658474Z 
2022-11-23T02:16:44.4658631Z Generating XML reports...
2022-11-23T02:16:44.4659102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021612.xml
2022-11-23T02:16:44.4659509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4659716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4660128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4660357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4660377Z 
2022-11-23T02:16:44.4660513Z Running tests...
2022-11-23T02:16:44.4660806Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4661155Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4661508Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4661761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51587
2022-11-23T02:16:44.4662014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51588
2022-11-23T02:16:44.4662421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4662630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4663043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4663267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4663667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4663880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4664278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4664501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4664779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4665054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4665534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4665973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4666238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4666504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4666775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.4667030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.4667458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4667886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4668162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:16:44.4668435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:16:44.4668856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.4669282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.4669574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpslgh_raa
2022-11-23T02:16:44.4669878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpslgh_raa/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4670148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4j61z4a0
2022-11-23T02:16:44.4670452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4j61z4a0/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4670592Z ok (10.214s)
2022-11-23T02:16:44.4670612Z 
2022-11-23T02:16:44.4670917Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4671066Z Ran 1 test in 10.214s
2022-11-23T02:16:44.4671085Z 
2022-11-23T02:16:44.4671259Z OK
2022-11-23T02:16:44.4671279Z 
2022-11-23T02:16:44.4671434Z Generating XML reports...
2022-11-23T02:16:44.4671923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021620.xml
2022-11-23T02:16:44.4672310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4672520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4672935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4673163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4673183Z 
2022-11-23T02:16:44.4673326Z Running tests...
2022-11-23T02:16:44.4673621Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4673968Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:16:44.4674296Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:16:44.4674548Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51710
2022-11-23T02:16:44.4674781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51711
2022-11-23T02:16:44.4675187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4675390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4675896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4676125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4676522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:16:44.4676735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:16:44.4677151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:16:44.4677360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:16:44.4677639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:16:44.4677916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:16:44.4678356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4678780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:16:44.4679042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:16:44.4679306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:16:44.4679579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:16:44.4679846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:16:44.4680261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4680688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:16:44.4680968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:16:44.4681240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:16:44.4681663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.4682145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:16:44.4682433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd51y1heo
2022-11-23T02:16:44.4682737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd51y1heo/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4683025Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4i96wrox
2022-11-23T02:16:44.4683314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4i96wrox/_remote_module_non_scriptable.py
2022-11-23T02:16:44.4683442Z ok (10.152s)
2022-11-23T02:16:44.4683462Z 
2022-11-23T02:16:44.4683766Z ----------------------------------------------------------------------
2022-11-23T02:16:44.4683913Z Ran 1 test in 10.153s
2022-11-23T02:16:44.4683933Z 
2022-11-23T02:16:44.4684060Z OK
2022-11-23T02:16:44.4684082Z 
2022-11-23T02:16:44.4684242Z Generating XML reports...
2022-11-23T02:16:44.4684731Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021633.xml
2022-11-23T02:16:44.4684750Z 
2022-11-23T02:16:44.4685236Z ##[endgroup]
2022-11-23T02:16:44.4685936Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_vzl_fg2e)
2022-11-23T02:16:44.4685959Z 
2022-11-23T02:16:44.4686122Z Shard 1: ucc should be run in 3
2022-11-23T02:16:44.4686432Z Running distributed/rpc/test_faulty_agent ... [2022-11-23 02:16:44.157558]
2022-11-23T02:16:44.4687054Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:16:44.157803]
2022-11-23T02:16:46.7485157Z 
2022-11-23T02:16:46.7485985Z Expand the folded group to see the log file of distributed/rpc/test_faulty_agent
2022-11-23T02:16:46.7487465Z ##[group]PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent_7pmiinri)
2022-11-23T02:16:46.7488138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjowhqn3m
2022-11-23T02:16:46.7488757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjowhqn3m/_remote_module_non_scriptable.py
2022-11-23T02:16:46.7489117Z 
2022-11-23T02:16:46.7489457Z ##[endgroup]
2022-11-23T02:16:46.7490252Z FINISHED PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent_7pmiinri)
2022-11-23T02:16:46.7490641Z 
2022-11-23T02:16:46.7490969Z Running distributed/pipeline/sync/test_stream ... [2022-11-23 02:16:46.748664]
2022-11-23T02:16:46.7492582Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-11-23 02:16:46.748932]
2022-11-23T02:16:52.2502296Z 
2022-11-23T02:16:52.2503038Z Expand the folded group to see the log file of distributed/pipeline/sync/test_stream
2022-11-23T02:16:52.2504127Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_al11g66g)
2022-11-23T02:16:52.2504716Z ============================= test session starts ==============================
2022-11-23T02:16:52.2505701Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:16:52.2506158Z cachedir: .pytest_cache
2022-11-23T02:16:52.2506836Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:16:52.2507317Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:16:52.2507710Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:16:52.2508348Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:16:52.2509061Z collecting ... collected 19 items
2022-11-23T02:16:52.2511738Z Running 19 items in this shard: test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view
2022-11-23T02:16:52.2514269Z 
2022-11-23T02:16:52.2514552Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [  5%]
2022-11-23T02:16:52.2515111Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda PASSED [ 10%]
2022-11-23T02:16:52.2515711Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%]
2022-11-23T02:16:52.2516297Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda PASSED [ 21%]
2022-11-23T02:16:52.2516876Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%]
2022-11-23T02:16:52.2517456Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda PASSED [ 31%]
2022-11-23T02:16:52.2518003Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%]
2022-11-23T02:16:52.2518560Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda PASSED [ 42%]
2022-11-23T02:16:52.2519104Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%]
2022-11-23T02:16:52.2519629Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda PASSED [ 52%]
2022-11-23T02:16:52.2520178Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%]
2022-11-23T02:16:52.2520720Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda PASSED [ 63%]
2022-11-23T02:16:52.2521274Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%]
2022-11-23T02:16:52.2521824Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda PASSED [ 73%]
2022-11-23T02:16:52.2522382Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu PASSED [ 78%]
2022-11-23T02:16:52.2522952Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda PASSED [ 84%]
2022-11-23T02:16:52.2523505Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%]
2022-11-23T02:16:52.2524076Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda PASSED [ 94%]
2022-11-23T02:16:52.2524733Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view PASSED [100%]
2022-11-23T02:16:52.2525059Z 
2022-11-23T02:16:52.2525252Z ============================== 19 passed in 2.52s ==============================
2022-11-23T02:16:52.2525484Z 
2022-11-23T02:16:52.2526175Z ##[endgroup]
2022-11-23T02:16:52.2526911Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_al11g66g)
2022-11-23T02:16:52.2527330Z 
2022-11-23T02:16:52.2527645Z Running distributed/pipeline/sync/test_phony ... [2022-11-23 02:16:52.250325]
2022-11-23T02:16:52.2528340Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-11-23 02:16:52.250591]
2022-11-23T02:16:54.7064990Z 
2022-11-23T02:16:54.7065520Z Expand the folded group to see the log file of distributed/pipeline/sync/test_phony
2022-11-23T02:16:54.7067016Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_mq0iy0kk)
2022-11-23T02:16:54.7067625Z ============================= test session starts ==============================
2022-11-23T02:16:54.7068303Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:16:54.7068730Z cachedir: .pytest_cache
2022-11-23T02:16:54.7069371Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:16:54.7069863Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:16:54.7070723Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:16:54.7071814Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:16:54.7072471Z collecting ... collected 4 items
2022-11-23T02:16:54.7073594Z Running 4 items in this shard: test/distributed/pipeline/sync/test_phony.py::test_phony_size, test/distributed/pipeline/sync/test_phony.py::test_phony_requires_grad, test/distributed/pipeline/sync/test_phony.py::test_cached_phony, test/distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function
2022-11-23T02:16:54.7074483Z 
2022-11-23T02:16:54.7074837Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED          [ 25%]
2022-11-23T02:16:54.7075555Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%]
2022-11-23T02:16:54.7076266Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED        [ 75%]
2022-11-23T02:16:54.7077001Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%]
2022-11-23T02:16:54.7077430Z 
2022-11-23T02:16:54.7077718Z ============================== 4 passed in 0.06s ===============================
2022-11-23T02:16:54.7078022Z 
2022-11-23T02:16:54.7078562Z ##[endgroup]
2022-11-23T02:16:54.7079565Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_mq0iy0kk)
2022-11-23T02:16:54.7080218Z 
2022-11-23T02:16:54.7080707Z Running distributed/pipeline/sync/test_dependency ... [2022-11-23 02:16:54.706637]
2022-11-23T02:16:54.7081911Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-11-23 02:16:54.706898]
2022-11-23T02:16:58.5308668Z 
2022-11-23T02:16:58.5309495Z Expand the folded group to see the log file of distributed/pipeline/sync/test_dependency
2022-11-23T02:16:58.5310594Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency_zfuuar93)
2022-11-23T02:16:58.5311219Z ============================= test session starts ==============================
2022-11-23T02:16:58.5311914Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:16:58.5312343Z cachedir: .pytest_cache
2022-11-23T02:16:58.5313251Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:16:58.5313746Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:16:58.5314154Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:16:58.5314800Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:16:58.5315323Z collecting ... collected 6 items
2022-11-23T02:16:58.5316307Z Running 6 items in this shard: test/distributed/pipeline/sync/test_dependency.py::test_fork_join, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_leak, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad
2022-11-23T02:16:58.5317144Z 
2022-11-23T02:16:58.5317402Z distributed/pipeline/sync/test_dependency.py::test_fork_join PASSED      [ 16%]
2022-11-23T02:16:58.5317922Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%]
2022-11-23T02:16:58.5318461Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%]
2022-11-23T02:16:58.5318982Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED      [ 66%]
2022-11-23T02:16:58.5319506Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%]
2022-11-23T02:16:58.5320167Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%]
2022-11-23T02:16:58.5320486Z 
2022-11-23T02:16:58.5320682Z ============================== 6 passed in 1.39s ===============================
2022-11-23T02:16:58.5320915Z 
2022-11-23T02:16:58.5321304Z ##[endgroup]
2022-11-23T02:16:58.5322024Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency_zfuuar93)
2022-11-23T02:16:58.5322468Z 
2022-11-23T02:16:58.5322798Z Running distributed/pipeline/sync/test_checkpoint ... [2022-11-23 02:16:58.530829]
2022-11-23T02:16:58.5323499Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-11-23 02:16:58.531080]
2022-11-23T02:17:02.3831729Z 
2022-11-23T02:17:02.3832304Z Expand the folded group to see the log file of distributed/pipeline/sync/test_checkpoint
2022-11-23T02:17:02.3833587Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_smssy74i)
2022-11-23T02:17:02.3834205Z ============================= test session starts ==============================
2022-11-23T02:17:02.3834900Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:17:02.3835334Z cachedir: .pytest_cache
2022-11-23T02:17:02.3835979Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:17:02.3836495Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:17:02.3836903Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:17:02.3837557Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:17:02.3838018Z collecting ... collected 9 items
2022-11-23T02:17:02.3839445Z Running 9 items in this shard: test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad, test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter, test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing, test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint, test/distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output
2022-11-23T02:17:02.3840909Z 
2022-11-23T02:17:02.3841191Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 11%]
2022-11-23T02:17:02.3841756Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda] PASSED [ 22%]
2022-11-23T02:17:02.3842311Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 33%]
2022-11-23T02:17:02.3842860Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 44%]
2022-11-23T02:17:02.3843436Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 55%]
2022-11-23T02:17:02.3843998Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda] PASSED [ 66%]
2022-11-23T02:17:02.3844581Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 77%]
2022-11-23T02:17:02.3845198Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 88%]
2022-11-23T02:17:02.3846065Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%]
2022-11-23T02:17:02.3846373Z 
2022-11-23T02:17:02.3846570Z ============================== 9 passed in 1.36s ===============================
2022-11-23T02:17:02.3846805Z 
2022-11-23T02:17:02.3847178Z ##[endgroup]
2022-11-23T02:17:02.3848038Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_smssy74i)
2022-11-23T02:17:02.3848481Z 
2022-11-23T02:17:02.3848842Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-11-23 02:17:02.383184]
2022-11-23T02:17:02.3849588Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-11-23 02:17:02.383446]
2022-11-23T02:17:04.8408573Z 
2022-11-23T02:17:04.8409383Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_verify_skippables
2022-11-23T02:17:04.8410923Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_r2rnynnl)
2022-11-23T02:17:04.8411687Z ============================= test session starts ==============================
2022-11-23T02:17:04.8412488Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:17:04.8413066Z cachedir: .pytest_cache
2022-11-23T02:17:04.8413781Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:17:04.8414514Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:17:04.8414952Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:17:04.8415800Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:17:04.8416285Z collecting ... collected 9 items
2022-11-23T02:17:04.8417960Z Running 9 items in this shard: test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated
2022-11-23T02:17:04.8419755Z 
2022-11-23T02:17:04.8420046Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%]
2022-11-23T02:17:04.8420655Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%]
2022-11-23T02:17:04.8421388Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%]
2022-11-23T02:17:04.8421993Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%]
2022-11-23T02:17:04.8422706Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%]
2022-11-23T02:17:04.8423422Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%]
2022-11-23T02:17:04.8424106Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%]
2022-11-23T02:17:04.8424881Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%]
2022-11-23T02:17:04.8425484Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%]
2022-11-23T02:17:04.8425878Z 
2022-11-23T02:17:04.8426190Z ============================== 9 passed in 0.05s ===============================
2022-11-23T02:17:04.8426434Z 
2022-11-23T02:17:04.8426819Z ##[endgroup]
2022-11-23T02:17:04.8427863Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_r2rnynnl)
2022-11-23T02:17:04.8428524Z 
2022-11-23T02:17:04.8429028Z Running distributed/pipeline/sync/skip/test_portal ... [2022-11-23 02:17:04.840830]
2022-11-23T02:17:04.8429775Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-11-23 02:17:04.841083]
2022-11-23T02:17:08.7115225Z 
2022-11-23T02:17:08.7115795Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_portal
2022-11-23T02:17:08.7117388Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_4qr9xdmo)
2022-11-23T02:17:08.7118017Z ============================= test session starts ==============================
2022-11-23T02:17:08.7118704Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:17:08.7119131Z cachedir: .pytest_cache
2022-11-23T02:17:08.7120102Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:17:08.7120623Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:17:08.7121015Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:17:08.7121662Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:17:08.7122125Z collecting ... collected 10 items
2022-11-23T02:17:08.7123582Z Running 10 items in this shard: test/distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad, test/distributed/pipeline/sync/skip/test_portal.py::test_use_grad, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1
2022-11-23T02:17:08.7124826Z 
2022-11-23T02:17:08.7125340Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device PASSED [ 10%]
2022-11-23T02:17:08.7126268Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED   [ 20%]
2022-11-23T02:17:08.7126827Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%]
2022-11-23T02:17:08.7127345Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED      [ 40%]
2022-11-23T02:17:08.7127880Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%]
2022-11-23T02:17:08.7128446Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%]
2022-11-23T02:17:08.7129006Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%]
2022-11-23T02:17:08.7129537Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%]
2022-11-23T02:17:08.7130086Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%]
2022-11-23T02:17:08.7130664Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%]
2022-11-23T02:17:08.7130982Z 
2022-11-23T02:17:08.7131193Z ============================== 10 passed in 1.39s ==============================
2022-11-23T02:17:08.7131428Z 
2022-11-23T02:17:08.7131798Z ##[endgroup]
2022-11-23T02:17:08.7132540Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_4qr9xdmo)
2022-11-23T02:17:08.7132975Z 
2022-11-23T02:17:08.7133419Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-11-23 02:17:08.711528]
2022-11-23T02:17:08.7134111Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-11-23 02:17:08.711786]
2022-11-23T02:17:16.2342665Z 
2022-11-23T02:17:16.2343578Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_gpipe
2022-11-23T02:17:16.2345632Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_wch631b3)
2022-11-23T02:17:16.2346728Z ============================= test session starts ==============================
2022-11-23T02:17:16.2347421Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:17:16.2347864Z cachedir: .pytest_cache
2022-11-23T02:17:16.2348512Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:17:16.2349008Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:17:16.2349407Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:17:16.2350047Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:17:16.2350518Z collecting ... collected 13 items
2022-11-23T02:17:16.2352813Z Running 13 items in this shard: test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip
2022-11-23T02:17:16.2354446Z 
2022-11-23T02:17:16.2354787Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] PASSED  [  7%]
2022-11-23T02:17:16.2355386Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] PASSED [ 15%]
2022-11-23T02:17:16.2355974Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] PASSED [ 23%]
2022-11-23T02:17:16.2356560Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%]
2022-11-23T02:17:16.2357157Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] PASSED [ 38%]
2022-11-23T02:17:16.2357758Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] PASSED [ 46%]
2022-11-23T02:17:16.2358341Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] PASSED [ 53%]
2022-11-23T02:17:16.2358946Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%]
2022-11-23T02:17:16.2359554Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] PASSED [ 69%]
2022-11-23T02:17:16.2360167Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] PASSED [ 76%]
2022-11-23T02:17:16.2360761Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] PASSED [ 84%]
2022-11-23T02:17:16.2361379Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%]
2022-11-23T02:17:16.2361903Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED      [100%]
2022-11-23T02:17:16.2362187Z 
2022-11-23T02:17:16.2362388Z ======================== 10 passed, 3 skipped in 4.71s =========================
2022-11-23T02:17:16.2362608Z 
2022-11-23T02:17:16.2362988Z ##[endgroup]
2022-11-23T02:17:16.2363807Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_wch631b3)
2022-11-23T02:17:16.2364252Z 
2022-11-23T02:17:16.2364558Z Running distributed/fsdp/test_shard_utils ... [2022-11-23 02:17:16.234309]
2022-11-23T02:17:16.2365297Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:16.234577]
2022-11-23T02:17:18.3982420Z 
2022-11-23T02:17:18.3983571Z Expand the folded group to see the log file of distributed/fsdp/test_shard_utils
2022-11-23T02:17:18.3985374Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_a0ec7y37)
2022-11-23T02:17:18.3986063Z 
2022-11-23T02:17:18.3986672Z ##[endgroup]
2022-11-23T02:17:18.3988026Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_a0ec7y37)
2022-11-23T02:17:18.3988713Z 
2022-11-23T02:17:18.3989286Z Running distributed/checkpoint/test_traverse ... [2022-11-23 02:17:18.398340]
2022-11-23T02:17:18.3990126Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:18.398693]
2022-11-23T02:17:22.4062389Z 
2022-11-23T02:17:22.4063243Z Expand the folded group to see the log file of distributed/checkpoint/test_traverse
2022-11-23T02:17:22.4064622Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_traverse_wenjny64)
2022-11-23T02:17:22.4065053Z 
2022-11-23T02:17:22.4065207Z Running tests...
2022-11-23T02:17:22.4065823Z ----------------------------------------------------------------------
2022-11-23T02:17:22.4066775Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_traverse
2022-11-23T02:17:22.4067322Z   test_get_element (__main__.TestTraverse) ... ok (1.617s)
2022-11-23T02:17:22.4067749Z   test_set_element (__main__.TestTraverse) ... ok (0.002s)
2022-11-23T02:17:22.4068220Z   test_traverse_doesnt_ignore_intermediate_collections (__main__.TestTraverse) ... ok (0.003s)
2022-11-23T02:17:22.4069014Z   test_traverse_nested_dict (__main__.TestTraverse) ... ok (0.001s)
2022-11-23T02:17:22.4069476Z   test_traverse_nested_list (__main__.TestTraverse) ... ok (0.002s)
2022-11-23T02:17:22.4069930Z   test_traverse_shallow (__main__.TestTraverse) ... ok (0.002s)
2022-11-23T02:17:22.4070374Z   test_traverse_with_ordered_dict (__main__.TestTraverse) ... ok (0.001s)
2022-11-23T02:17:22.4070643Z 
2022-11-23T02:17:22.4070963Z ----------------------------------------------------------------------
2022-11-23T02:17:22.4071375Z Ran 7 tests in 1.630s
2022-11-23T02:17:22.4071579Z 
2022-11-23T02:17:22.4071693Z OK
2022-11-23T02:17:22.4071868Z 
2022-11-23T02:17:22.4072039Z Generating XML reports...
2022-11-23T02:17:22.4072723Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_traverse/TEST-TestTraverse-20221123021720.xml
2022-11-23T02:17:22.4073108Z 
2022-11-23T02:17:22.4073489Z ##[endgroup]
2022-11-23T02:17:22.4074194Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_traverse_wenjny64)
2022-11-23T02:17:22.4074618Z 
2022-11-23T02:17:22.4074948Z Running distributed/_shard/test_replicated_tensor ... [2022-11-23 02:17:22.406213]
2022-11-23T02:17:22.4075747Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_replicated_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:22.406477]
2022-11-23T02:17:24.5582021Z 
2022-11-23T02:17:24.5582885Z Expand the folded group to see the log file of distributed/_shard/test_replicated_tensor
2022-11-23T02:17:24.5584247Z ##[group]PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_9z5u98xv)
2022-11-23T02:17:24.5584721Z 
2022-11-23T02:17:24.5585090Z ##[endgroup]
2022-11-23T02:17:24.5585943Z FINISHED PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_9z5u98xv)
2022-11-23T02:17:24.5586403Z 
2022-11-23T02:17:24.5586709Z Running distributed/test_nccl ... [2022-11-23 02:17:24.558245]
2022-11-23T02:17:24.5587754Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:24.558502]
2022-11-23T02:17:29.6439638Z 
2022-11-23T02:17:29.6440140Z Expand the folded group to see the log file of distributed/test_nccl
2022-11-23T02:17:29.6441143Z ##[group]PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_do2r1e2u)
2022-11-23T02:17:29.6441505Z 
2022-11-23T02:17:29.6441696Z Running tests...
2022-11-23T02:17:29.6442272Z ----------------------------------------------------------------------
2022-11-23T02:17:29.6442873Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl
2022-11-23T02:17:29.6443392Z   test_all_gather_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (1.122s)
2022-11-23T02:17:29.6443851Z   test_all_gather_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-11-23T02:17:29.6444315Z   test_all_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-11-23T02:17:29.6444777Z   test_all_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-11-23T02:17:29.6445245Z   test_broadcast_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-11-23T02:17:29.6445983Z   test_broadcast_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-11-23T02:17:29.6446466Z   test_collective_errors_cuda (__main__.TestNCCLCUDA) ... ok (0.002s)
2022-11-23T02:17:29.6446941Z   test_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-11-23T02:17:29.6447374Z   test_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-11-23T02:17:29.6447837Z   test_reduce_scatter_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-11-23T02:17:29.6448314Z   test_reduce_scatter_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-11-23T02:17:29.6449022Z   test_unique_id_cuda (__main__.TestNCCLCUDA) ... ok (0.001s)
2022-11-23T02:17:29.6449255Z 
2022-11-23T02:17:29.6449577Z ----------------------------------------------------------------------
2022-11-23T02:17:29.6449986Z Ran 12 tests in 1.156s
2022-11-23T02:17:29.6450192Z 
2022-11-23T02:17:29.6450327Z OK
2022-11-23T02:17:29.6450499Z 
2022-11-23T02:17:29.6450646Z Generating XML reports...
2022-11-23T02:17:29.6451287Z Generated XML report: test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20221123021728.xml
2022-11-23T02:17:29.6451645Z 
2022-11-23T02:17:29.6452042Z ##[endgroup]
2022-11-23T02:17:29.6452650Z FINISHED PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_do2r1e2u)
2022-11-23T02:17:29.6453010Z 
2022-11-23T02:17:29.6453332Z Running distributed/_composable/test_contract ... [2022-11-23 02:17:29.643996]
2022-11-23T02:17:29.6454106Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_contract.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:29.644261]
2022-11-23T02:17:33.5026504Z 
2022-11-23T02:17:33.5027495Z Expand the folded group to see the log file of distributed/_composable/test_contract
2022-11-23T02:17:33.5029637Z ##[group]PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_contract_0qn9t2ct)
2022-11-23T02:17:33.5030494Z 
2022-11-23T02:17:33.5030773Z Running tests...
2022-11-23T02:17:33.5031479Z ----------------------------------------------------------------------
2022-11-23T02:17:33.5032743Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_contract
2022-11-23T02:17:33.5033667Z   test_add_hooks (__main__.TestContract) ... ok (1.608s)
2022-11-23T02:17:33.5034089Z   test_modify_fqn (__main__.TestContract) ... ok (0.002s)
2022-11-23T02:17:33.5034495Z   test_state (__main__.TestContract) ... ok (0.002s)
2022-11-23T02:17:33.5034730Z 
2022-11-23T02:17:33.5035042Z ----------------------------------------------------------------------
2022-11-23T02:17:33.5035441Z Ran 3 tests in 1.613s
2022-11-23T02:17:33.5035635Z 
2022-11-23T02:17:33.5035763Z OK
2022-11-23T02:17:33.5035929Z 
2022-11-23T02:17:33.5036084Z Generating XML reports...
2022-11-23T02:17:33.5036733Z Generated XML report: test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20221123021731.xml
2022-11-23T02:17:33.5037114Z 
2022-11-23T02:17:33.5037505Z ##[endgroup]
2022-11-23T02:17:33.5038185Z FINISHED PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_contract_0qn9t2ct)
2022-11-23T02:17:33.5038572Z 
2022-11-23T02:17:33.5038881Z Running distributed/checkpoint/test_utils ... [2022-11-23 02:17:33.502650]
2022-11-23T02:17:33.5039644Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:33.502908]
2022-11-23T02:17:37.4523992Z 
2022-11-23T02:17:37.4524584Z Expand the folded group to see the log file of distributed/checkpoint/test_utils
2022-11-23T02:17:37.4525639Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_utils_oq9sjkqd)
2022-11-23T02:17:37.4526668Z 
2022-11-23T02:17:37.4526892Z Running tests...
2022-11-23T02:17:37.4527848Z ----------------------------------------------------------------------
2022-11-23T02:17:37.4528960Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_utils
2022-11-23T02:17:37.4529751Z   test_flat_data (__main__.TestMedatadaIndex) ... ok (1.653s)
2022-11-23T02:17:37.4530209Z   test_index_hint_ignored_on_equals (__main__.TestMedatadaIndex) ... ok (0.001s)
2022-11-23T02:17:37.4530696Z   test_index_hint_ignored_on_hash (__main__.TestMedatadaIndex) ... ok (0.001s)
2022-11-23T02:17:37.4531166Z   test_init_convert_offset (__main__.TestMedatadaIndex) ... ok (0.001s)
2022-11-23T02:17:37.4531896Z   test_sharded_tensor_lookup (__main__.TestMedatadaIndex) ... ok (0.003s)
2022-11-23T02:17:37.4532170Z 
2022-11-23T02:17:37.4532483Z ----------------------------------------------------------------------
2022-11-23T02:17:37.4532878Z Ran 5 tests in 1.659s
2022-11-23T02:17:37.4533078Z 
2022-11-23T02:17:37.4533201Z OK
2022-11-23T02:17:37.4533346Z 
2022-11-23T02:17:37.4533496Z Generating XML reports...
2022-11-23T02:17:37.4534179Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_utils/TEST-TestMedatadaIndex-20221123021735.xml
2022-11-23T02:17:37.4534591Z 
2022-11-23T02:17:37.4534991Z ##[endgroup]
2022-11-23T02:17:37.4535651Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_utils_oq9sjkqd)
2022-11-23T02:17:37.4536052Z 
2022-11-23T02:17:37.4536370Z Running distributed/elastic/utils/logging_test ... [2022-11-23 02:17:37.452411]
2022-11-23T02:17:37.4537148Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:37.452703]
2022-11-23T02:17:41.4824480Z 
2022-11-23T02:17:41.4825224Z Expand the folded group to see the log file of distributed/elastic/utils/logging_test
2022-11-23T02:17:41.4826335Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_w1t0trk7)
2022-11-23T02:17:41.4826765Z 
2022-11-23T02:17:41.4826916Z Running tests...
2022-11-23T02:17:41.4827790Z ----------------------------------------------------------------------
2022-11-23T02:17:41.4828461Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test
2022-11-23T02:17:41.4828994Z   test_derive_module_name (__main__.LoggingTest) ... ok (1.687s)
2022-11-23T02:17:41.4829424Z   test_logger_name (__main__.LoggingTest) ... ok (0.002s)
2022-11-23T02:17:41.4829681Z 
2022-11-23T02:17:41.4829983Z ----------------------------------------------------------------------
2022-11-23T02:17:41.4830372Z Ran 2 tests in 1.690s
2022-11-23T02:17:41.4830575Z 
2022-11-23T02:17:41.4830706Z OK
2022-11-23T02:17:41.4830879Z 
2022-11-23T02:17:41.4831042Z Generating XML reports...
2022-11-23T02:17:41.4831712Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20221123021739.xml
2022-11-23T02:17:41.4832105Z 
2022-11-23T02:17:41.4832487Z ##[endgroup]
2022-11-23T02:17:41.4833204Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_w1t0trk7)
2022-11-23T02:17:41.4833625Z 
2022-11-23T02:17:41.4833934Z Running distributed/fsdp/test_checkpoint_wrapper ... [2022-11-23 02:17:41.482428]
2022-11-23T02:17:41.4834725Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:41.482791]
2022-11-23T02:17:45.9041406Z 
2022-11-23T02:17:45.9042219Z Expand the folded group to see the log file of distributed/fsdp/test_checkpoint_wrapper
2022-11-23T02:17:45.9043328Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_1jp6znar)
2022-11-23T02:17:45.9043758Z 
2022-11-23T02:17:45.9043908Z Running tests...
2022-11-23T02:17:45.9044467Z ----------------------------------------------------------------------
2022-11-23T02:17:45.9045140Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper
2022-11-23T02:17:45.9046018Z   test_apply_activation_checkpointing (__main__.CheckpointWrapperTest)
2022-11-23T02:17:45.9046873Z Ensures that `apply_activation_checkpointing` can be used ... ok (1.722s)
2022-11-23T02:17:45.9047715Z   test_checkpoint_wrapper_cpu_offload (__main__.CheckpointWrapperTest) ... ok (0.404s)
2022-11-23T02:17:45.9048640Z   test_checkpoint_wrapper_kwarg_support (__main__.CheckpointWrapperTest) ... ok (0.009s)
2022-11-23T02:17:45.9049153Z   test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest)
2022-11-23T02:17:45.9050355Z Tests that using checkpoint_wrapper or the functional ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79510 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-11-23T02:17:45.9051226Z   test_forward_missing_attributes (__main__.CheckpointWrapperTest) ... ok (0.001s)
2022-11-23T02:17:45.9051688Z   test_fqn (__main__.CheckpointWrapperTest) ... ok (0.001s)
2022-11-23T02:17:45.9052204Z   test_load_activation_checkpointed_module (__main__.CheckpointWrapperTest) ... ok (0.003s)
2022-11-23T02:17:45.9052511Z 
2022-11-23T02:17:45.9052818Z ----------------------------------------------------------------------
2022-11-23T02:17:45.9053200Z Ran 7 tests in 2.143s
2022-11-23T02:17:45.9053395Z 
2022-11-23T02:17:45.9053541Z OK (skipped=1)
2022-11-23T02:17:45.9053724Z 
2022-11-23T02:17:45.9053879Z Generating XML reports...
2022-11-23T02:17:45.9054588Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20221123021743.xml
2022-11-23T02:17:45.9055011Z 
2022-11-23T02:17:45.9055375Z ##[endgroup]
2022-11-23T02:17:45.9056196Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_1jp6znar)
2022-11-23T02:17:45.9056623Z 
2022-11-23T02:17:45.9056924Z Running distributed/test_data_parallel ... [2022-11-23 02:17:45.904154]
2022-11-23T02:17:45.9057650Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:45.904534]
2022-11-23T02:17:54.7373917Z 
2022-11-23T02:17:54.7374719Z Expand the folded group to see the log file of distributed/test_data_parallel
2022-11-23T02:17:54.7376666Z ##[group]PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel_yjjbphfl)
2022-11-23T02:17:54.7377462Z 
2022-11-23T02:17:54.7377726Z Running tests...
2022-11-23T02:17:54.7378691Z ----------------------------------------------------------------------
2022-11-23T02:17:54.7379427Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel
2022-11-23T02:17:54.7380369Z   test_autocast (__main__.TestDataParallel) ... ok (1.871s)
2022-11-23T02:17:54.7381268Z   test_data_parallel (__main__.TestDataParallel) ... ok (0.092s)
2022-11-23T02:17:54.7382221Z   test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... ok (0.013s)
2022-11-23T02:17:54.7382743Z   test_data_parallel_complex (__main__.TestDataParallel) ... ok (0.062s)
2022-11-23T02:17:54.7383243Z   test_data_parallel_device_args (__main__.TestDataParallel) ... ok (0.006s)
2022-11-23T02:17:54.7383726Z   test_data_parallel_function_deletion (__main__.TestDataParallel) ... ok (0.006s)
2022-11-23T02:17:54.7384724Z   test_data_parallel_lazy_linear (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2022-11-23T02:17:54.7385540Z   warnings.warn('Lazy modules are a new feature under heavy development '
2022-11-23T02:17:54.7385944Z ok (0.002s)
2022-11-23T02:17:54.7386317Z   test_data_parallel_model_device (__main__.TestDataParallel)
2022-11-23T02:17:54.7386752Z Test device[0] check at forward time. ... ok (0.035s)
2022-11-23T02:17:54.7387214Z   test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... ok (0.101s)
2022-11-23T02:17:54.7387707Z   test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... ok (0.005s)
2022-11-23T02:17:54.7388941Z   test_data_parallel_multiple_input (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/comm.py:231: UserWarning: Using -1 to represent CPU tensor is deprecated. Please use a device object or string instead, e.g., "cpu".
2022-11-23T02:17:54.7389581Z   warnings.warn(
2022-11-23T02:17:54.7389894Z ok (0.022s)
2022-11-23T02:17:54.7390281Z   test_data_parallel_nested_input (__main__.TestDataParallel) ... ok (0.003s)
2022-11-23T02:17:54.7390775Z   test_data_parallel_nested_output (__main__.TestDataParallel) ... ok (0.006s)
2022-11-23T02:17:54.7391266Z   test_data_parallel_no_grad (__main__.TestDataParallel) ... ok (0.004s)
2022-11-23T02:17:54.7391724Z   test_data_parallel_rnn (__main__.TestDataParallel) ... ok (0.966s)
2022-11-23T02:17:54.7392202Z   test_data_parallel_small_back (__main__.TestDataParallel) ... ok (0.003s)
2022-11-23T02:17:54.7392682Z   test_data_parallel_sparse (__main__.TestDataParallel) ... ok (0.012s)
2022-11-23T02:17:54.7393622Z   test_gather_cpu (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
2022-11-23T02:17:54.7394369Z   warnings.warn('Was asked to gather along dimension 0, but all '
2022-11-23T02:17:54.7394749Z ok (0.044s)
2022-11-23T02:17:54.7395152Z   test_gather_different_len_dicts (__main__.TestDataParallel) ... ok (0.001s)
2022-11-23T02:17:54.7395616Z   test_gather_gpu (__main__.TestDataParallel) ... ok (0.044s)
2022-11-23T02:17:54.7396158Z   test_parallel_apply (__main__.TestDataParallel) ... ok (0.005s)
2022-11-23T02:17:54.7396646Z   test_parallel_apply_autocast (__main__.TestDataParallel) ... ok (0.006s)
2022-11-23T02:17:54.7397145Z   test_parallel_apply_passes_exception (__main__.TestDataParallel) ... ok (0.002s)
2022-11-23T02:17:54.7397630Z   test_parameter_list_dict_replica (__main__.TestDataParallel) ... ok (0.007s)
2022-11-23T02:17:54.7398104Z   test_replicate (__main__.TestDataParallel) ... ok (0.004s)
2022-11-23T02:17:54.7398559Z   test_replicate_buffers (__main__.TestDataParallel) ... ok (0.003s)
2022-11-23T02:17:54.7399012Z   test_save_replica_module (__main__.TestDataParallel) ... ok (0.003s)
2022-11-23T02:17:54.7399469Z   test_scatter_cpu (__main__.TestDataParallel) ... ok (0.018s)
2022-11-23T02:17:54.7399912Z   test_scatter_gpu (__main__.TestDataParallel) ... ok (0.019s)
2022-11-23T02:17:54.7400398Z   test_strided_grad_layout (__main__.TestDataParallel) ... ok (0.982s)
2022-11-23T02:17:54.7400834Z   test_zero_grad (__main__.TestDataParallel) ... ok (0.008s)
2022-11-23T02:17:54.7401357Z   test_data_parallel_module_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7401942Z   test_data_parallel_module_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7402522Z   test_data_parallel_module_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2022-11-23T02:17:54.7403101Z   test_data_parallel_module_kwargs_only_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7403712Z   test_data_parallel_module_kwargs_only_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7404325Z   test_data_parallel_module_kwargs_only_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7404954Z   test_data_parallel_module_kwargs_only_empty_dict_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7405580Z   test_data_parallel_module_kwargs_only_empty_dict_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7406427Z   test_data_parallel_module_kwargs_only_empty_dict_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7407063Z   test_data_parallel_module_kwargs_only_empty_list_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7407772Z   test_data_parallel_module_kwargs_only_empty_list_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7408408Z   test_data_parallel_module_kwargs_only_empty_list_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7409039Z   test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7409678Z   test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7410297Z   test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2022-11-23T02:17:54.7410637Z 
2022-11-23T02:17:54.7411000Z ----------------------------------------------------------------------
2022-11-23T02:17:54.7411412Z Ran 46 tests in 4.432s
2022-11-23T02:17:54.7411614Z 
2022-11-23T02:17:54.7411751Z OK
2022-11-23T02:17:54.7411926Z 
2022-11-23T02:17:54.7412074Z Generating XML reports...
2022-11-23T02:17:54.7412737Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20221123021749.xml
2022-11-23T02:17:54.7413613Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20221123021749.xml
2022-11-23T02:17:54.7414039Z 
2022-11-23T02:17:54.7414423Z ##[endgroup]
2022-11-23T02:17:54.7415081Z FINISHED PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel_yjjbphfl)
2022-11-23T02:17:54.7415545Z 
2022-11-23T02:17:54.7415874Z Running distributed/fsdp/test_fsdp_traversal ... [2022-11-23 02:17:54.737516]
2022-11-23T02:17:54.7416646Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:17:54.737849]
2022-11-23T02:18:01.9114753Z 
2022-11-23T02:18:01.9115302Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_traversal
2022-11-23T02:18:01.9117295Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_fd46oyt5)
2022-11-23T02:18:01.9118150Z 
2022-11-23T02:18:01.9118374Z Running tests...
2022-11-23T02:18:01.9118977Z ----------------------------------------------------------------------
2022-11-23T02:18:01.9119613Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal
2022-11-23T02:18:01.9120457Z   test_fsdp_modules (__main__.TestTraversal) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:18:01.9121370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53191
2022-11-23T02:18:01.9122347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53192
2022-11-23T02:18:01.9123717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:01.9124250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:01.9124907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:01.9125480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:01.9126586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:01.9127097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:01.9127752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:01.9128298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:01.9128806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:01.9129682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:01.9130457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:01.9131269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:01.9131889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:01.9132465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:01.9132924Z dist init r=0, world=2
2022-11-23T02:18:01.9133249Z dist init r=1, world=2
2022-11-23T02:18:01.9133576Z ok (4.833s)
2022-11-23T02:18:01.9133768Z 
2022-11-23T02:18:01.9134091Z ----------------------------------------------------------------------
2022-11-23T02:18:01.9134511Z Ran 1 test in 4.834s
2022-11-23T02:18:01.9134702Z 
2022-11-23T02:18:01.9134833Z OK
2022-11-23T02:18:01.9135008Z 
2022-11-23T02:18:01.9135172Z Generating XML reports...
2022-11-23T02:18:01.9135866Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221123021756.xml
2022-11-23T02:18:01.9136268Z 
2022-11-23T02:18:01.9136660Z ##[endgroup]
2022-11-23T02:18:01.9137381Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_fd46oyt5)
2022-11-23T02:18:01.9137814Z 
2022-11-23T02:18:01.9138139Z Running distributed/fsdp/test_fsdp_uneven ... [2022-11-23 02:18:01.911467]
2022-11-23T02:18:01.9139035Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:18:01.911727]
2022-11-23T02:18:09.5732569Z 
2022-11-23T02:18:09.5733118Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_uneven
2022-11-23T02:18:09.5734190Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_uneven_0p1_derc)
2022-11-23T02:18:09.5734598Z 
2022-11-23T02:18:09.5734729Z Running tests...
2022-11-23T02:18:09.5735300Z ----------------------------------------------------------------------
2022-11-23T02:18:09.5735932Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven
2022-11-23T02:18:09.5736435Z   test_one_iteration (__main__.TestUnevenParamShard)
2022-11-23T02:18:09.5736917Z Test FSDP with uneven divide of parameter shards. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:18:09.5737461Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53305
2022-11-23T02:18:09.5737986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53306
2022-11-23T02:18:09.5738693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:09.5739216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:09.5739871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:09.5740411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:09.5741046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:09.5741568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:09.5742216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:09.5742751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:09.5743252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:09.5743819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:09.5744820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:09.5745560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:09.5746155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:09.5746705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:09.5747135Z dist init r=1, world=2
2022-11-23T02:18:09.5747436Z dist init r=0, world=2
2022-11-23T02:18:09.5747747Z ok (5.328s)
2022-11-23T02:18:09.5747926Z 
2022-11-23T02:18:09.5748230Z ----------------------------------------------------------------------
2022-11-23T02:18:09.5748609Z Ran 1 test in 5.328s
2022-11-23T02:18:09.5748801Z 
2022-11-23T02:18:09.5748931Z OK
2022-11-23T02:18:09.5749105Z 
2022-11-23T02:18:09.5749258Z Generating XML reports...
2022-11-23T02:18:09.5749930Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20221123021803.xml
2022-11-23T02:18:09.5750331Z 
2022-11-23T02:18:09.5750709Z ##[endgroup]
2022-11-23T02:18:09.5751375Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_uneven (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_uneven_0p1_derc)
2022-11-23T02:18:09.5751769Z 
2022-11-23T02:18:09.5752104Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-11-23 02:18:09.573280]
2022-11-23T02:18:09.5752984Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:18:09.573544]
2022-11-23T02:18:17.1800059Z 
2022-11-23T02:18:17.1800706Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_wrapping
2022-11-23T02:18:17.1802453Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_ryim3a00)
2022-11-23T02:18:17.1803010Z 
2022-11-23T02:18:17.1803374Z Running tests...
2022-11-23T02:18:17.1803950Z ----------------------------------------------------------------------
2022-11-23T02:18:17.1804616Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2022-11-23T02:18:17.1805155Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2022-11-23T02:18:17.1805948Z This test simulates wrapping the module after training to run inference. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:18:17.1806518Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53423
2022-11-23T02:18:17.1807042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53424
2022-11-23T02:18:17.1807758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:17.1808273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:17.1808931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:17.1809508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:17.1810162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:17.1810663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:17.1811315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:17.1811851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:17.1812357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:17.1813272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:17.1814047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:17.1814876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:17.1815484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:17.1816048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:17.1817470Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:18:17.1818405Z   warnings.warn(
2022-11-23T02:18:17.1819700Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:18:17.1820715Z   warnings.warn(
2022-11-23T02:18:17.1821047Z dist init r=0, world=2
2022-11-23T02:18:17.1821386Z dist init r=1, world=2
2022-11-23T02:18:17.1821714Z ok (5.300s)
2022-11-23T02:18:17.1821892Z 
2022-11-23T02:18:17.1822218Z ----------------------------------------------------------------------
2022-11-23T02:18:17.1822643Z Ran 1 test in 5.300s
2022-11-23T02:18:17.1822847Z 
2022-11-23T02:18:17.1822967Z OK
2022-11-23T02:18:17.1823147Z 
2022-11-23T02:18:17.1823286Z Generating XML reports...
2022-11-23T02:18:17.1824018Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221123021811.xml
2022-11-23T02:18:17.1824465Z 
2022-11-23T02:18:17.1824866Z ##[endgroup]
2022-11-23T02:18:17.1825597Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_ryim3a00)
2022-11-23T02:18:17.1826055Z 
2022-11-23T02:18:17.1826421Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2022-11-23 02:18:17.180057]
2022-11-23T02:18:17.1827273Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:18:17.180323]
2022-11-23T02:18:25.7986306Z 
2022-11-23T02:18:25.7987322Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding
2022-11-23T02:18:25.7988506Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_mlugo6xf)
2022-11-23T02:18:25.7988961Z 
2022-11-23T02:18:25.7989112Z Running tests...
2022-11-23T02:18:25.7989932Z ----------------------------------------------------------------------
2022-11-23T02:18:25.7991166Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding
2022-11-23T02:18:25.7991815Z   test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:18:25.7992377Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53541
2022-11-23T02:18:25.7992877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53542
2022-11-23T02:18:25.7993694Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53543
2022-11-23T02:18:25.7994198Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53544
2022-11-23T02:18:25.7994913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:25.7995424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:25.7996081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:25.7996791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:25.7998132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:25.7999033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:25.8000296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:25.8001275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:25.8001928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:25.8002450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:25.8003103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:25.8003635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:25.8004418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:25.8004952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:25.8005601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:25.8006548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:25.8007037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:25.8007581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:25.8008155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:25.8008732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:25.8009171Z skip: Need at least 4 CUDA devices (3.910s)
2022-11-23T02:18:25.8009734Z   test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53677
2022-11-23T02:18:25.8010349Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53678
2022-11-23T02:18:25.8010847Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53679
2022-11-23T02:18:25.8011367Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53680
2022-11-23T02:18:25.8012057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:25.8012577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:25.8013210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:25.8013750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:25.8014399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:25.8014902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:25.8015551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:25.8016203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:25.8016848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:25.8017346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:25.8017986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:25.8018522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:25.8019172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:25.8019668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:25.8020305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:25.8020838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:25.8021329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:25.8021873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:25.8022405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:25.8022944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:25.8023384Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:18:25.8023612Z 
2022-11-23T02:18:25.8023995Z ----------------------------------------------------------------------
2022-11-23T02:18:25.8024402Z Ran 2 tests in 6.320s
2022-11-23T02:18:25.8024599Z 
2022-11-23T02:18:25.8024725Z OK (skipped=2)
2022-11-23T02:18:25.8024909Z 
2022-11-23T02:18:25.8025063Z Generating XML reports...
2022-11-23T02:18:25.8025779Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20221123021819.xml
2022-11-23T02:18:25.8026206Z 
2022-11-23T02:18:25.8026590Z ##[endgroup]
2022-11-23T02:18:25.8027337Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_mlugo6xf)
2022-11-23T02:18:25.8027776Z 
2022-11-23T02:18:25.8028111Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2022-11-23 02:18:25.798716]
2022-11-23T02:18:25.8028912Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:18:25.798982]
2022-11-23T02:18:34.4061910Z 
2022-11-23T02:18:34.4062656Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_chunk
2022-11-23T02:18:34.4063809Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk_by2em8v5)
2022-11-23T02:18:34.4064272Z 
2022-11-23T02:18:34.4064424Z Running tests...
2022-11-23T02:18:34.4065064Z ----------------------------------------------------------------------
2022-11-23T02:18:34.4065893Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk
2022-11-23T02:18:34.4066746Z   test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:18:34.4067641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53848
2022-11-23T02:18:34.4068645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53849
2022-11-23T02:18:34.4069572Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53850
2022-11-23T02:18:34.4070442Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53851
2022-11-23T02:18:34.4071744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:34.4073074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:34.4074332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:34.4075390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:34.4076584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:34.4077530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:34.4078774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:34.4079769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:34.4080942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:34.4081878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:34.4083134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:34.4084119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:34.4085323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:34.4086653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:34.4088097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:34.4089102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:34.4090066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:34.4091124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:34.4092169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:34.4093196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:34.4094060Z skip: Need at least 4 CUDA devices (4.006s)
2022-11-23T02:18:34.4095113Z   test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53984
2022-11-23T02:18:34.4096281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53985
2022-11-23T02:18:34.4097253Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53986
2022-11-23T02:18:34.4098235Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53987
2022-11-23T02:18:34.4099546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:34.4100490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:34.4101700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:34.4102701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:34.4103959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:34.4104900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:34.4106148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:34.4107176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:34.4108483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:34.4109620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:34.4110883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:34.4111890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:34.4113134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:34.4114109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:34.4115374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:34.4116335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:34.4117306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:34.4118332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:34.4119361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:34.4120361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:34.4121229Z skip: Need at least 4 CUDA devices (2.310s)
2022-11-23T02:18:34.4121635Z 
2022-11-23T02:18:34.4122198Z ----------------------------------------------------------------------
2022-11-23T02:18:34.4122890Z Ran 2 tests in 6.316s
2022-11-23T02:18:34.4123250Z 
2022-11-23T02:18:34.4123501Z OK (skipped=2)
2022-11-23T02:18:34.4123835Z 
2022-11-23T02:18:34.4124243Z Generating XML reports...
2022-11-23T02:18:34.4125653Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20221123021827.xml
2022-11-23T02:18:34.4126781Z 
2022-11-23T02:18:34.4127451Z ##[endgroup]
2022-11-23T02:18:34.4128836Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk_by2em8v5)
2022-11-23T02:18:34.4129685Z 
2022-11-23T02:18:34.4130257Z Running distributed/test_c10d_error_logger ... [2022-11-23 02:18:34.406245]
2022-11-23T02:18:34.4131683Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_error_logger.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:18:34.406520]
2022-11-23T02:18:43.8880581Z 
2022-11-23T02:18:43.8881749Z Expand the folded group to see the log file of distributed/test_c10d_error_logger
2022-11-23T02:18:43.8883663Z ##[group]PRINTING LOG FILE of distributed/test_c10d_error_logger (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_error_logger_ylq88_60)
2022-11-23T02:18:43.8884118Z 
2022-11-23T02:18:43.8884367Z Running tests...
2022-11-23T02:18:43.8885021Z ----------------------------------------------------------------------
2022-11-23T02:18:43.8885643Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_error_logger
2022-11-23T02:18:43.8886855Z   test_exception_handler_with_dist (__main__.C10dErrorLoggerTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:18:43.8887829Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54155
2022-11-23T02:18:43.8888473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54156
2022-11-23T02:18:43.8889170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:43.8889691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:43.8890344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:43.8890868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:43.8891514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:43.8892322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:43.8892989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:43.8893555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:43.8894088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:43.8894673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:18:43.8895245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:43.8895830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:18:43.8896598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:43.8897405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:18:43.8897879Z ok (4.846s)
2022-11-23T02:18:43.8898404Z   test_get_or_create_logger (__main__.C10dErrorLoggerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54234
2022-11-23T02:18:43.8899032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54235
2022-11-23T02:18:43.8899724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:43.8900370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:43.8901061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:43.8901629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:43.8902291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:43.8902840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:43.8903514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:43.8904063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:43.8904593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:43.8905161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:43.8905585Z ok (2.307s)
2022-11-23T02:18:43.8905755Z 
2022-11-23T02:18:43.8906072Z ----------------------------------------------------------------------
2022-11-23T02:18:43.8906480Z Ran 2 tests in 7.153s
2022-11-23T02:18:43.8906686Z 
2022-11-23T02:18:43.8906812Z OK
2022-11-23T02:18:43.8906982Z 
2022-11-23T02:18:43.8907125Z Generating XML reports...
2022-11-23T02:18:43.8907871Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_error_logger/TEST-C10dErrorLoggerTest-20221123021836.xml
2022-11-23T02:18:43.8908278Z 
2022-11-23T02:18:43.8908673Z ##[endgroup]
2022-11-23T02:18:43.8909349Z FINISHED PRINTING LOG FILE of distributed/test_c10d_error_logger (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_error_logger_ylq88_60)
2022-11-23T02:18:43.8909763Z 
2022-11-23T02:18:43.8910130Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2022-11-23 02:18:43.888064]
2022-11-23T02:18:43.8910986Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:18:43.888336]
2022-11-23T02:18:55.1749943Z 
2022-11-23T02:18:55.1750779Z Expand the folded group to see the log file of distributed/_shard/sharding_spec/test_sharding_spec
2022-11-23T02:18:55.1751901Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_z8002m6_)
2022-11-23T02:18:55.1753517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpta2qq56e
2022-11-23T02:18:55.1754657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpta2qq56e/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1755316Z 
2022-11-23T02:18:55.1755563Z Running tests...
2022-11-23T02:18:55.1756497Z ----------------------------------------------------------------------
2022-11-23T02:18:55.1757597Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec
2022-11-23T02:18:55.1758681Z   test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:18:55.1759807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54337
2022-11-23T02:18:55.1760549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54338
2022-11-23T02:18:55.1761201Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54339
2022-11-23T02:18:55.1762009Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54340
2022-11-23T02:18:55.1763083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:55.1763894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:55.1764873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:55.1766450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:55.1767437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:55.1768260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:55.1769444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:55.1770397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:55.1771435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:55.1772189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:55.1772850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:55.1773384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:55.1774048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:55.1774572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:55.1775224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:55.1775758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:55.1776302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbbx4lebj
2022-11-23T02:18:55.1776925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbbx4lebj/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1777521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpar8xuwur
2022-11-23T02:18:55.1778143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpar8xuwur/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1778743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4h6xhpf
2022-11-23T02:18:55.1779352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4h6xhpf/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1779942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjk95v1o3
2022-11-23T02:18:55.1780700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjk95v1o3/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1781281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:55.1781830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:55.1782354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:55.1782892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:55.1783318Z ok (4.092s)
2022-11-23T02:18:55.1783719Z   test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec)
2022-11-23T02:18:55.1784298Z Test custom spec can be invoked from the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54473
2022-11-23T02:18:55.1784872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54474
2022-11-23T02:18:55.1785410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54475
2022-11-23T02:18:55.1785917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54476
2022-11-23T02:18:55.1786605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:55.1787132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:55.1787768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:55.1788374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:55.1789041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:55.1789561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:55.1790263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:55.1790795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:55.1791449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:55.1791970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:55.1792602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:55.1793147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:55.1793799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:55.1794318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:55.1794945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:55.1795491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:55.1796033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdptpih_u
2022-11-23T02:18:55.1796654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdptpih_u/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1797229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:55.1797806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhtr3xec
2022-11-23T02:18:55.1798418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhtr3xec/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1799007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74qoqd0c
2022-11-23T02:18:55.1799614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74qoqd0c/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1800290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhw1lzy6
2022-11-23T02:18:55.1800899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhw1lzy6/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1801466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:55.1802011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:55.1802552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:55.1803005Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:18:55.1803463Z   test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec)
2022-11-23T02:18:55.1804049Z Test sharded_tensor.ones(...) with the custom ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54609
2022-11-23T02:18:55.1804631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54610
2022-11-23T02:18:55.1805137Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54611
2022-11-23T02:18:55.1805646Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54612
2022-11-23T02:18:55.1806857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:55.1807368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:55.1808029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:55.1808676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:55.1809348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:55.1809850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:55.1810505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:55.1811046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:55.1811695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:55.1812197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:55.1812838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:55.1813379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:55.1814015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:18:55.1814531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:18:55.1815174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:18:55.1815717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:18:55.1816241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxwbjpfr1
2022-11-23T02:18:55.1816864Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxwbjpfr1/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1817466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpij2vsqzr
2022-11-23T02:18:55.1818058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpij2vsqzr/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1818664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoedbofzk
2022-11-23T02:18:55.1819278Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoedbofzk/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1819857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:18:55.1820482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:18:55.1821017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:18:55.1821589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyz1zff5j
2022-11-23T02:18:55.1822199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyz1zff5j/_remote_module_non_scriptable.py
2022-11-23T02:18:55.1822760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:18:55.1823229Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:18:55.1823667Z   test_check_overlapping (__main__.TestShardingSpec) ... ok (0.003s)
2022-11-23T02:18:55.1824127Z   test_chunked_sharding_spec (__main__.TestShardingSpec) ... ok (0.012s)
2022-11-23T02:18:55.1824594Z   test_device_placement (__main__.TestShardingSpec) ... ok (0.007s)
2022-11-23T02:18:55.1825075Z   test_enumerable_sharding_spec (__main__.TestShardingSpec) ... ok (0.007s)
2022-11-23T02:18:55.1825563Z   test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s)
2022-11-23T02:18:55.1826026Z   test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-11-23T02:18:55.1826487Z   test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s)
2022-11-23T02:18:55.1826978Z   test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.009s)
2022-11-23T02:18:55.1827270Z 
2022-11-23T02:18:55.1827571Z ----------------------------------------------------------------------
2022-11-23T02:18:55.1828038Z Ran 11 tests in 8.957s
2022-11-23T02:18:55.1828247Z 
2022-11-23T02:18:55.1828393Z OK (skipped=2)
2022-11-23T02:18:55.1828586Z 
2022-11-23T02:18:55.1828747Z Generating XML reports...
2022-11-23T02:18:55.1829458Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20221123021845.xml
2022-11-23T02:18:55.1830370Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20221123021845.xml
2022-11-23T02:18:55.1830766Z 
2022-11-23T02:18:55.1831184Z ##[endgroup]
2022-11-23T02:18:55.1831914Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_z8002m6_)
2022-11-23T02:18:55.1832351Z 
2022-11-23T02:18:55.1832688Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2022-11-23 02:18:55.175028]
2022-11-23T02:18:55.1833492Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:18:55.175336]
2022-11-23T02:19:06.3291859Z 
2022-11-23T02:19:06.3292428Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T02:19:06.3293665Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_si844_q1)
2022-11-23T02:19:06.3294155Z 
2022-11-23T02:19:06.3294307Z Running tests...
2022-11-23T02:19:06.3294906Z ----------------------------------------------------------------------
2022-11-23T02:19:06.3295562Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear
2022-11-23T02:19:06.3296194Z   test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:19:06.3296785Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54780
2022-11-23T02:19:06.3297323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54781
2022-11-23T02:19:06.3298046Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54782
2022-11-23T02:19:06.3298565Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54783
2022-11-23T02:19:06.3299562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:06.3300079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:06.3300746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:06.3301301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:06.3301966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:06.3302474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:06.3303129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:06.3303671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:06.3304323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:06.3304847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:06.3305501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:06.3306094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:06.3306739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:06.3307372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:06.3308041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:06.3308585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:06.3309083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:06.3309637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:06.3310188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:06.3310711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:06.3311175Z skip: Need at least 4 CUDA devices (4.027s)
2022-11-23T02:19:06.3311763Z   test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54916
2022-11-23T02:19:06.3312388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54917
2022-11-23T02:19:06.3312893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54918
2022-11-23T02:19:06.3313407Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54919
2022-11-23T02:19:06.3314108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:06.3314622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:06.3315279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:06.3315827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:06.3316488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:06.3317000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:06.3317659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:06.3318209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:06.3318845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:06.3319443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:06.3320094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:06.3320634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:06.3321265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:06.3321784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:06.3322434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:06.3322960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:06.3323471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:06.3324019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:06.3324560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:06.3325120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:06.3325592Z skip: Need at least 4 CUDA devices (2.412s)
2022-11-23T02:19:06.3326460Z   test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55052
2022-11-23T02:19:06.3327188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55053
2022-11-23T02:19:06.3327708Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55054
2022-11-23T02:19:06.3328220Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55055
2022-11-23T02:19:06.3328917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:06.3329448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:06.3330083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:06.3330627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:06.3331282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:06.3331785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:06.3332439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:06.3332980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:06.3333630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:06.3334137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:06.3334785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:06.3335320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:06.3335955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:06.3336470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:06.3337152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:06.3337686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:06.3338179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:06.3338843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:06.3339382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:06.3339923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:06.3340369Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:19:06.3340598Z 
2022-11-23T02:19:06.3340910Z ----------------------------------------------------------------------
2022-11-23T02:19:06.3341314Z Ran 3 tests in 8.850s
2022-11-23T02:19:06.3341514Z 
2022-11-23T02:19:06.3341646Z OK (skipped=3)
2022-11-23T02:19:06.3341855Z 
2022-11-23T02:19:06.3342022Z Generating XML reports...
2022-11-23T02:19:06.3342768Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20221123021857.xml
2022-11-23T02:19:06.3343205Z 
2022-11-23T02:19:06.3343593Z ##[endgroup]
2022-11-23T02:19:06.3344334Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_si844_q1)
2022-11-23T02:19:06.3344768Z 
2022-11-23T02:19:06.3345113Z Running distributed/_tensor/parallel/test_parallelize_api ... [2022-11-23 02:19:06.329336]
2022-11-23T02:19:06.3345923Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_parallelize_api.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:19:06.329603]
2022-11-23T02:19:20.0754558Z 
2022-11-23T02:19:20.0755646Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_parallelize_api
2022-11-23T02:19:20.0757299Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_parallelize_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_parallelize_api_bxlciex6)
2022-11-23T02:19:20.0757977Z 
2022-11-23T02:19:20.0758173Z Running tests...
2022-11-23T02:19:20.0759222Z ----------------------------------------------------------------------
2022-11-23T02:19:20.0760231Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api
2022-11-23T02:19:20.0761063Z   test_creat_1d_device_mesh (__main__.TensorParallelAPITests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:19:20.0761615Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55223
2022-11-23T02:19:20.0762144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55224
2022-11-23T02:19:20.0762668Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55225
2022-11-23T02:19:20.0763174Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55226
2022-11-23T02:19:20.0763879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0764408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0765066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0765602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0766529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0767056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0767688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0768246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0768902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0769424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0770057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0770749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0771407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0771928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0772562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0773108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0773620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:20.0774154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:20.0774902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:20.0775461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:20.0775930Z skip: Need at least 4 CUDA devices (4.024s)
2022-11-23T02:19:20.0776492Z   test_creat_1d_device_mesh_error (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55359
2022-11-23T02:19:20.0777112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55360
2022-11-23T02:19:20.0777637Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55361
2022-11-23T02:19:20.0778238Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55362
2022-11-23T02:19:20.0778950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0779480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0780137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0780678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0781331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0781850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0782501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0783029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0783682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0784203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0784831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0785384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0786034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0786552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0787181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0787719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0788233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:20.0788773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:20.0789324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:20.0789924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:20.0790394Z skip: Need at least 4 CUDA devices (2.409s)
2022-11-23T02:19:20.0790936Z   test_parallelize_mlp (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55495
2022-11-23T02:19:20.0791550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55496
2022-11-23T02:19:20.0792079Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55497
2022-11-23T02:19:20.0792584Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55498
2022-11-23T02:19:20.0793285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0793812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0794464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0795001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0795657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0796176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0796825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0797353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0798068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0798593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0799220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0799767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0800420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0800937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0801574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0802118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0802635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:20.0803171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:20.0803720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:20.0804249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:20.0804760Z skip: Need at least 4 CUDA devices (2.413s)
2022-11-23T02:19:20.0805313Z   test_parallelize_mlp_error (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55631
2022-11-23T02:19:20.0806104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55632
2022-11-23T02:19:20.0806652Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55633
2022-11-23T02:19:20.0807170Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55634
2022-11-23T02:19:20.0807852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0808381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0809032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0809675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0810317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0810837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0811478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0812001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0812661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0813182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0813828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0814346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0815002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:20.0815520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:20.0816169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:20.0816688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:20.0817199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:20.0817821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:20.0818359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:20.0818894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:20.0819356Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:19:20.0819593Z 
2022-11-23T02:19:20.0819907Z ----------------------------------------------------------------------
2022-11-23T02:19:20.0820299Z Ran 4 tests in 11.358s
2022-11-23T02:19:20.0820504Z 
2022-11-23T02:19:20.0820648Z OK (skipped=4)
2022-11-23T02:19:20.0820843Z 
2022-11-23T02:19:20.0821006Z Generating XML reports...
2022-11-23T02:19:20.0821720Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api/TEST-TensorParallelAPITests-20221123021908.xml
2022-11-23T02:19:20.0822153Z 
2022-11-23T02:19:20.0822555Z ##[endgroup]
2022-11-23T02:19:20.0823303Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_parallelize_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_parallelize_api_bxlciex6)
2022-11-23T02:19:20.0823751Z 
2022-11-23T02:19:20.0824038Z Running distributed/_tensor/test_math_ops ... [2022-11-23 02:19:20.075559]
2022-11-23T02:19:20.0824793Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:19:20.075821]
2022-11-23T02:19:34.1138444Z 
2022-11-23T02:19:34.1138965Z Expand the folded group to see the log file of distributed/_tensor/test_math_ops
2022-11-23T02:19:34.1139988Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_math_ops_a1pt1hx1)
2022-11-23T02:19:34.1140385Z 
2022-11-23T02:19:34.1140534Z Running tests...
2022-11-23T02:19:34.1141096Z ----------------------------------------------------------------------
2022-11-23T02:19:34.1141728Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_math_ops
2022-11-23T02:19:34.1142281Z   test_softmax_fwd (__main__.DistMathOpsTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:19:34.1142793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55802
2022-11-23T02:19:34.1143603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55803
2022-11-23T02:19:34.1144304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:34.1144818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:34.1145449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:34.1145996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:34.1146675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:34.1147193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:34.1147814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:34.1148356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:34.1148867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:34.1149410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:34.1149949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:34.1150509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:34.1151233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:34.1152087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:34.1152553Z ok (4.921s)
2022-11-23T02:19:34.1153031Z   test_softmax_with_bwd (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55883
2022-11-23T02:19:34.1153612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55884
2022-11-23T02:19:34.1154279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:34.1154791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:34.1155431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:34.1155968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:34.1156604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:34.1157116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:34.1157756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:34.1158277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:34.1158775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:34.1159329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:34.1159882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:34.1160414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:34.1161139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:34.1161899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:34.1162360Z ok (3.410s)
2022-11-23T02:19:34.1162808Z   test_sum (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55968
2022-11-23T02:19:34.1163434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55969
2022-11-23T02:19:34.1164108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:34.1164607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:34.1165248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:34.1166087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:34.1166763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:34.1167260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:34.1167899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:34.1168433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:34.1168917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:34.1169472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:19:34.1170020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:34.1170568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:19:34.1171357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:34.1172121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:19:34.1172583Z ok (3.310s)
2022-11-23T02:19:34.1172760Z 
2022-11-23T02:19:34.1173067Z ----------------------------------------------------------------------
2022-11-23T02:19:34.1173446Z Ran 3 tests in 11.642s
2022-11-23T02:19:34.1173636Z 
2022-11-23T02:19:34.1173755Z OK
2022-11-23T02:19:34.1173918Z 
2022-11-23T02:19:34.1174074Z Generating XML reports...
2022-11-23T02:19:34.1174706Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_math_ops/TEST-DistMathOpsTest-20221123021922.xml
2022-11-23T02:19:34.1175083Z 
2022-11-23T02:19:34.1175458Z ##[endgroup]
2022-11-23T02:19:34.1176107Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_math_ops_a1pt1hx1)
2022-11-23T02:19:34.1176492Z 
2022-11-23T02:19:34.1176819Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2022-11-23 02:19:34.113904]
2022-11-23T02:19:34.1177624Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:19:34.114249]
2022-11-23T02:19:50.1226369Z 
2022-11-23T02:19:50.1227124Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_tensor_ops
2022-11-23T02:19:50.1228337Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_in4seux8)
2022-11-23T02:19:50.1228817Z 
2022-11-23T02:19:50.1228962Z Running tests...
2022-11-23T02:19:50.1229538Z ----------------------------------------------------------------------
2022-11-23T02:19:50.1230212Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops
2022-11-23T02:19:50.1230784Z   test_clone (__main__.TestTensorOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:19:50.1231301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56084
2022-11-23T02:19:50.1231806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56085
2022-11-23T02:19:50.1232594Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56086
2022-11-23T02:19:50.1233091Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56087
2022-11-23T02:19:50.1233790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1234300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1234942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1235488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1236149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1236647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1237296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1237845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1238463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1238977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1239612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1240243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1240891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1241400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1242037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1242559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1243066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:50.1243604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:50.1244138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:50.1244659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:50.1245110Z skip: Need at least 4 CUDA devices (4.062s)
2022-11-23T02:19:50.1245635Z   test_deep_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56220
2022-11-23T02:19:50.1246566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56221
2022-11-23T02:19:50.1247062Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56222
2022-11-23T02:19:50.1247578Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56223
2022-11-23T02:19:50.1248260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1248780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1249403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1249940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1250592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1251105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1251722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1252365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1253012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1253508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1254146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1254674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1255325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1255814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1256452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1256984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1257468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:50.1258009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:50.1258531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:50.1259070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:50.1259507Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:19:50.1260086Z   test_detach (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56356
2022-11-23T02:19:50.1279061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56357
2022-11-23T02:19:50.1280027Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56358
2022-11-23T02:19:50.1280951Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56359
2022-11-23T02:19:50.1282214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1283138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1284282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1285209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1286713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1287572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1288740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1289794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1290952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1291815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1292907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1293854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1294990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1295899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1297063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1298017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1299085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:50.1300052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:50.1301018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:50.1301978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:50.1302828Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:19:50.1303755Z   test_inplace_copy (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56492
2022-11-23T02:19:50.1304771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56493
2022-11-23T02:19:50.1305665Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56494
2022-11-23T02:19:50.1306579Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56495
2022-11-23T02:19:50.1307728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1308635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1309760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1310792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1312469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1319348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1320621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1321655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1322895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1323850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1325069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1326412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1327644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1328600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1329803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1330779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1331669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:50.1332637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:50.1333586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:50.1334550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:50.1335275Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:19:50.1335786Z   test_set_requires_grad (__main__.TestTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56628
2022-11-23T02:19:50.1336348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56629
2022-11-23T02:19:50.1336841Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56630
2022-11-23T02:19:50.1337327Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56631
2022-11-23T02:19:50.1337991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1338625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1339247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1339763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1340377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1340866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1341483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1341992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1342612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1343103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1343715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1344227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1344841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:19:50.1345337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:19:50.1346045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:19:50.1346592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:19:50.1347077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:19:50.1347616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:19:50.1348158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:19:50.1348685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:19:50.1349148Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:19:50.1349377Z 
2022-11-23T02:19:50.1349685Z ----------------------------------------------------------------------
2022-11-23T02:19:50.1350089Z Ran 5 tests in 13.702s
2022-11-23T02:19:50.1350266Z 
2022-11-23T02:19:50.1350413Z OK (skipped=5)
2022-11-23T02:19:50.1350602Z 
2022-11-23T02:19:50.1350763Z Generating XML reports...
2022-11-23T02:19:50.1351452Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20221123021936.xml
2022-11-23T02:19:50.1351851Z 
2022-11-23T02:19:50.1352242Z ##[endgroup]
2022-11-23T02:19:50.1352989Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_in4seux8)
2022-11-23T02:19:50.1353436Z 
2022-11-23T02:19:50.1353780Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2022-11-23 02:19:50.122702]
2022-11-23T02:19:50.1354592Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:19:50.122995]
2022-11-23T02:20:06.0375712Z 
2022-11-23T02:20:06.0376291Z Expand the folded group to see the log file of distributed/_shard/sharding_plan/test_sharding_plan
2022-11-23T02:20:06.0377521Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_33s35lt3)
2022-11-23T02:20:06.0378305Z 
2022-11-23T02:20:06.0378472Z Running tests...
2022-11-23T02:20:06.0379337Z ----------------------------------------------------------------------
2022-11-23T02:20:06.0380012Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan
2022-11-23T02:20:06.0380618Z   test_custom_sharding_planner (__main__.TestShardingPlan) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:20:06.0381246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56799
2022-11-23T02:20:06.0381762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56800
2022-11-23T02:20:06.0382278Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56801
2022-11-23T02:20:06.0382783Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56802
2022-11-23T02:20:06.0384153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0385093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0386175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0387127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0388369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0389391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0390607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0391838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0393060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0394009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0395209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0396214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0397413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0398334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0399561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0400553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0401550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:06.0402543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:06.0403546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:06.0404597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:06.0405442Z skip: Need at least 4 CUDA devices (3.945s)
2022-11-23T02:20:06.0406912Z   test_reshard_to_ddp_sharding_plan (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56935
2022-11-23T02:20:06.0408073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56936
2022-11-23T02:20:06.0409054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56937
2022-11-23T02:20:06.0410008Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56938
2022-11-23T02:20:06.0411355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0412337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0413592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0414786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0416054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0417014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0418211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0419220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0420487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0421452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0422669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0423696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0424936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0425869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0427125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0428131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0429276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:06.0430293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:06.0431306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:06.0432323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:06.0433186Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:20:06.0434207Z   test_shard_module_sub_process_group (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57071
2022-11-23T02:20:06.0435358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57072
2022-11-23T02:20:06.0436336Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57073
2022-11-23T02:20:06.0437310Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57074
2022-11-23T02:20:06.0438633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0439621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0440848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0441865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0443127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0444091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0445292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0446585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0447860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0448814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0450042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0451254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0452491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0453461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0454646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0455623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0456578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:06.0457563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:06.0458575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:06.0459562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:06.0460375Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:20:06.0461364Z   test_sharding_plan_errors (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57207
2022-11-23T02:20:06.0462472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57208
2022-11-23T02:20:06.0463440Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57209
2022-11-23T02:20:06.0464385Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57210
2022-11-23T02:20:06.0465896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0466887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0468129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0469136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0470369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0471341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0472568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0473609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0474871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0475838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0477077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0478067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0479334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0480314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0481540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0482542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0483468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:06.0484470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:06.0485487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:06.0486882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:06.0487740Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:20:06.0489057Z   test_sharding_plan_simple_megatron (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57343
2022-11-23T02:20:06.0490187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57344
2022-11-23T02:20:06.0491155Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57345
2022-11-23T02:20:06.0492099Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57346
2022-11-23T02:20:06.0493448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0494434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0495647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0496604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0497866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0498880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0500192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0501179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0502487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0503658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0504900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0505915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0507206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:06.0508211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:06.0509455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:06.0510483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:06.0511415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:20:06.0512418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:06.0513446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:06.0514455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:20:06.0515278Z skip: Need at least 4 CUDA devices (2.412s)
2022-11-23T02:20:06.0515711Z 
2022-11-23T02:20:06.0516259Z ----------------------------------------------------------------------
2022-11-23T02:20:06.0516958Z Ran 5 tests in 13.589s
2022-11-23T02:20:06.0517307Z 
2022-11-23T02:20:06.0517549Z OK (skipped=5)
2022-11-23T02:20:06.0517885Z 
2022-11-23T02:20:06.0518140Z Generating XML reports...
2022-11-23T02:20:06.0519476Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20221123021952.xml
2022-11-23T02:20:06.0520256Z 
2022-11-23T02:20:06.0520940Z ##[endgroup]
2022-11-23T02:20:06.0522350Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_33s35lt3)
2022-11-23T02:20:06.0523181Z 
2022-11-23T02:20:06.0523781Z Running distributed/fsdp/test_fsdp_memory ... [2022-11-23 02:20:06.037746]
2022-11-23T02:20:06.0525228Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:20:06.038069]
2022-11-23T02:20:22.1257212Z 
2022-11-23T02:20:22.1257919Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_memory
2022-11-23T02:20:22.1259031Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_le9yb77h)
2022-11-23T02:20:22.1259451Z 
2022-11-23T02:20:22.1259606Z Running tests...
2022-11-23T02:20:22.1260187Z ----------------------------------------------------------------------
2022-11-23T02:20:22.1260825Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_memory
2022-11-23T02:20:22.1261404Z   test_fsdp_memory_ckpt_ckpt (__main__.TestFSDPMemory) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:20:22.1261955Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57514
2022-11-23T02:20:22.1262468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57515
2022-11-23T02:20:22.1263223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:22.1263758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:22.1264417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:22.1264972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:22.1265610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:22.1266383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:22.1267061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:22.1267589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:22.1268135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:22.1268707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:22.1269445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:22.1270201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:22.1270804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:22.1271361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:22.1272713Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:20:22.1273590Z   warnings.warn(
2022-11-23T02:20:22.1274802Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:20:22.1275666Z   warnings.warn(
2022-11-23T02:20:22.1275992Z dist init r=1, world=2
2022-11-23T02:20:22.1276323Z dist init r=0, world=2
2022-11-23T02:20:22.1276627Z ok (7.983s)
2022-11-23T02:20:22.1277132Z   test_fsdp_memory_ckpt_no_ckpt (__main__.TestFSDPMemory) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57627
2022-11-23T02:20:22.1277845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57628
2022-11-23T02:20:22.1278526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:22.1279051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:22.1279706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:22.1280259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:22.1280901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:22.1281431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:22.1282081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:22.1282629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:22.1283142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:22.1283720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:22.1284455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:22.1285261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:22.1286148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:22.1286705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:22.1288073Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:20:22.1288940Z   warnings.warn(
2022-11-23T02:20:22.1290166Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:20:22.1291019Z   warnings.warn(
2022-11-23T02:20:22.1291350Z dist init r=1, world=2
2022-11-23T02:20:22.1291683Z dist init r=0, world=2
2022-11-23T02:20:22.1291988Z ok (5.815s)
2022-11-23T02:20:22.1292177Z 
2022-11-23T02:20:22.1292489Z ----------------------------------------------------------------------
2022-11-23T02:20:22.1292900Z Ran 2 tests in 13.798s
2022-11-23T02:20:22.1293101Z 
2022-11-23T02:20:22.1293214Z OK
2022-11-23T02:20:22.1293391Z 
2022-11-23T02:20:22.1293556Z Generating XML reports...
2022-11-23T02:20:22.1294218Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20221123022007.xml
2022-11-23T02:20:22.1294600Z 
2022-11-23T02:20:22.1294983Z ##[endgroup]
2022-11-23T02:20:22.1295662Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_memory_le9yb77h)
2022-11-23T02:20:22.1296058Z 
2022-11-23T02:20:22.1296392Z Running distributed/fsdp/test_fsdp_tp_integration ... [2022-11-23 02:20:22.125793]
2022-11-23T02:20:22.1297302Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:20:22.126068]
2022-11-23T02:20:42.4756388Z 
2022-11-23T02:20:42.4758986Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_tp_integration
2022-11-23T02:20:42.4760124Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration__f70jink)
2022-11-23T02:20:42.4760558Z 
2022-11-23T02:20:42.4760708Z Running tests...
2022-11-23T02:20:42.4766130Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4766806Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration
2022-11-23T02:20:42.4767357Z   test_fsdp_tp_checkpoint_integration (__main__.TestTPFSDPIntegration)
2022-11-23T02:20:42.4767905Z Tests checkpointing for TP + FSDP integration. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:20:42.4768449Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57775
2022-11-23T02:20:42.4768966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57776
2022-11-23T02:20:42.4769913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4770834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4772307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4772907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4773616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4774168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4774839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4775412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4775975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.4776560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.4777750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:42.4779341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:42.4780157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4780704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4781129Z dist init r=0, world=2
2022-11-23T02:20:42.4781587Z dist init r=1, world=2
2022-11-23T02:20:42.4782209Z skip: Need at least 4 CUDA devices (4.864s)
2022-11-23T02:20:42.4783168Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2022-11-23T02:20:42.4784820Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57854
2022-11-23T02:20:42.4785473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57855
2022-11-23T02:20:42.4786174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4786680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4787386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4788084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4788744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4789268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4789893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4790434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4790963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.4791531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.4792246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:42.4793012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:42.4793606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4794128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4794551Z dist init r=1, world=2
2022-11-23T02:20:42.4794867Z dist init r=0, world=2
2022-11-23T02:20:42.4795222Z skip: Need at least 4 CUDA devices (3.310s)
2022-11-23T02:20:42.4795791Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2022-11-23T02:20:42.4796623Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57933
2022-11-23T02:20:42.4797234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57934
2022-11-23T02:20:42.4797902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4798425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4799125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4799666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4800295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4800817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4801460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4801997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4802496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.4803061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.4803787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:42.4804532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:42.4805126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4806078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4806609Z dist init r=0, world=2
2022-11-23T02:20:42.4806908Z dist init r=1, world=2
2022-11-23T02:20:42.4807273Z skip: Need at least 4 CUDA devices (3.211s)
2022-11-23T02:20:42.4807813Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2022-11-23T02:20:42.4808738Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58012
2022-11-23T02:20:42.4809351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58013
2022-11-23T02:20:42.4810031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4810557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4811194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4811738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4812392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4812915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4813552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4814093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4814626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.4815179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.4816012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:42.4816791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:42.4817389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4817917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4818346Z dist init r=1, world=2
2022-11-23T02:20:42.4818670Z dist init r=0, world=2
2022-11-23T02:20:42.4819013Z skip: Need at least 4 CUDA devices (3.311s)
2022-11-23T02:20:42.4819549Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2022-11-23T02:20:42.4820370Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58091
2022-11-23T02:20:42.4820987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58092
2022-11-23T02:20:42.4821656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4822179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4822832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4823384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4824023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:20:42.4824542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:20:42.4825185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:20:42.4825707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:20:42.4826242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:20:42.4826817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:20:42.4827546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:42.4828358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:20:42.4828953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:20:42.4829495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:20:42.4829909Z dist init r=0, world=2
2022-11-23T02:20:42.4830236Z dist init r=1, world=2
2022-11-23T02:20:42.4830599Z skip: Need at least 4 CUDA devices (3.311s)
2022-11-23T02:20:42.4830832Z 
2022-11-23T02:20:42.4831144Z ----------------------------------------------------------------------
2022-11-23T02:20:42.4831534Z Ran 5 tests in 18.007s
2022-11-23T02:20:42.4831732Z 
2022-11-23T02:20:42.4831878Z OK (skipped=5)
2022-11-23T02:20:42.4832069Z 
2022-11-23T02:20:42.4832230Z Generating XML reports...
2022-11-23T02:20:42.4832917Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20221123022024.xml
2022-11-23T02:20:42.4833334Z 
2022-11-23T02:20:42.4833744Z ##[endgroup]
2022-11-23T02:20:42.4834455Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration__f70jink)
2022-11-23T02:20:42.4834869Z 
2022-11-23T02:20:42.4835174Z Running distributed/_composable/test_fully_shard ... [2022-11-23 02:20:42.475734]
2022-11-23T02:20:42.4836013Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_fully_shard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:20:42.476009]
2022-11-23T02:21:03.3245161Z 
2022-11-23T02:21:03.3246023Z Expand the folded group to see the log file of distributed/_composable/test_fully_shard
2022-11-23T02:21:03.3248042Z ##[group]PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_fully_shard_r4e7mj5h)
2022-11-23T02:21:03.3248873Z 
2022-11-23T02:21:03.3249069Z Running tests...
2022-11-23T02:21:03.3250086Z ----------------------------------------------------------------------
2022-11-23T02:21:03.3251097Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_fully_shard
2022-11-23T02:21:03.3251875Z   test_auto_wrap_policy (__main__.TestFSDPInitialization)
2022-11-23T02:21:03.3252905Z Tests passing an ``auto_wrap_policy``. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:03.3253755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58205
2022-11-23T02:21:03.3254300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58206
2022-11-23T02:21:03.3257120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:03.3257687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:03.3258392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:03.3258950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:03.3259597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:03.3260128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:03.3260780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:03.3261344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:03.3261863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:03.3262439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:03.3263178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:03.3264256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:03.3264944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:03.3265514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:03.3265963Z dist init r=0, world=2
2022-11-23T02:21:03.3266297Z dist init r=1, world=2
2022-11-23T02:21:03.3266604Z ok (4.851s)
2022-11-23T02:21:03.3266999Z   test_device_id (__main__.TestFSDPInitialization)
2022-11-23T02:21:03.3267557Z Tests passing a ``device_id``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58284
2022-11-23T02:21:03.3268154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58285
2022-11-23T02:21:03.3268864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:03.3269429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:03.3270118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:03.3270675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:03.3271372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:03.3271922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:03.3272713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:03.3273285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:03.3273839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:03.3274445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:03.3275201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:03.3276016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:03.3276644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:03.3277224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:03.3277662Z dist init r=1, world=2
2022-11-23T02:21:03.3277991Z dist init r=0, world=2
2022-11-23T02:21:03.3278317Z ok (3.309s)
2022-11-23T02:21:03.3278709Z   test_materialize_meta_module (__main__.TestFSDPInitialization)
2022-11-23T02:21:03.3279437Z Tests materializing a meta-device module. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58363
2022-11-23T02:21:03.3280060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58364
2022-11-23T02:21:03.3280781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:03.3281319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:03.3282000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:03.3282579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:03.3283257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:03.3283811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:03.3284490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:03.3285137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:03.3286061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:03.3286695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:03.3287474Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:03.3288291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:03.3288908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:03.3289480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:03.3289931Z dist init r=0, world=2
2022-11-23T02:21:03.3290265Z dist init r=1, world=2
2022-11-23T02:21:03.3290591Z ok (3.311s)
2022-11-23T02:21:03.3290991Z   test_sync_module_states (__main__.TestFSDPInitialization)
2022-11-23T02:21:03.3291561Z Tests passing ``sync_module_states=True``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58442
2022-11-23T02:21:03.3292178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58443
2022-11-23T02:21:03.3292908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:03.3293461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:03.3294224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:03.3294817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:03.3295514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:03.3296074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:03.3296745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:03.3297364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:03.3297921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:03.3298503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:03.3299276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:03.3300096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:03.3300724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:03.3301290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:03.3301746Z dist init r=0, world=2
2022-11-23T02:21:03.3302086Z dist init r=1, world=2
2022-11-23T02:21:03.3302399Z ok (3.209s)
2022-11-23T02:21:03.3302763Z   test_training (__main__.TestFSDPRuntime)
2022-11-23T02:21:03.3303327Z Tests training (forward, backward, optimizer). ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58521
2022-11-23T02:21:03.3303949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58522
2022-11-23T02:21:03.3304664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:03.3305225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:03.3305905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:03.3306560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:03.3307258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:03.3307835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:03.3308515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:03.3309075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:03.3309633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:03.3310242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:03.3311011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:03.3311813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:03.3312445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:03.3313026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:03.3313463Z dist init r=1, world=2
2022-11-23T02:21:03.3313803Z dist init r=0, world=2
2022-11-23T02:21:03.3314131Z ok (3.811s)
2022-11-23T02:21:03.3314325Z 
2022-11-23T02:21:03.3314646Z ----------------------------------------------------------------------
2022-11-23T02:21:03.3315115Z Ran 5 tests in 18.492s
2022-11-23T02:21:03.3315337Z 
2022-11-23T02:21:03.3315466Z OK
2022-11-23T02:21:03.3315644Z 
2022-11-23T02:21:03.3315811Z Generating XML reports...
2022-11-23T02:21:03.3316535Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPInitialization-20221123022044.xml
2022-11-23T02:21:03.3317456Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPRuntime-20221123022044.xml
2022-11-23T02:21:03.3317867Z 
2022-11-23T02:21:03.3318409Z ##[endgroup]
2022-11-23T02:21:03.3319146Z FINISHED PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_fully_shard_r4e7mj5h)
2022-11-23T02:21:03.3319565Z 
2022-11-23T02:21:03.3319887Z Running distributed/test_c10d_spawn_ucc ... [2022-11-23 02:21:03.324653]
2022-11-23T02:21:03.3320712Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:21:03.324986]
2022-11-23T02:21:49.2868746Z 
2022-11-23T02:21:49.2869470Z Expand the folded group to see the log file of distributed/test_c10d_spawn_ucc
2022-11-23T02:21:49.2870571Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_ucc_arovbp1u)
2022-11-23T02:21:49.2871266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37phm93j
2022-11-23T02:21:49.2871887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37phm93j/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2872961Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2873339Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:21:49.2873739Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:21:49.2878729Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsUcc testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_reduce>]>
2022-11-23T02:21:49.2880145Z test_all_gather (__main__.TestDistributedNNFunctionsUcc)
2022-11-23T02:21:49.2880888Z test_all_to_all (__main__.TestDistributedNNFunctionsUcc)
2022-11-23T02:21:49.2881395Z test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc)
2022-11-23T02:21:49.2881899Z test_allreduce (__main__.TestDistributedNNFunctionsUcc)
2022-11-23T02:21:49.2882399Z test_broadcast (__main__.TestDistributedNNFunctionsUcc)
2022-11-23T02:21:49.2882996Z test_reduce (__main__.TestDistributedNNFunctionsUcc)
2022-11-23T02:21:49.2884428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2885499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2886512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2887550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2888515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpax96cvpn
2022-11-23T02:21:49.2889651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpax96cvpn/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2890438Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2890808Z 
2022-11-23T02:21:49.2891062Z Running tests...
2022-11-23T02:21:49.2891974Z ----------------------------------------------------------------------
2022-11-23T02:21:49.2892916Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T02:21:49.2893685Z   test_all_gather (__main__.TestDistributedNNFunctionsUcc) ... skip: runs into illegal memory access on first assertEqual check when run locally (0.001s)
2022-11-23T02:21:49.2894124Z 
2022-11-23T02:21:49.2894432Z ----------------------------------------------------------------------
2022-11-23T02:21:49.2894832Z Ran 1 test in 0.001s
2022-11-23T02:21:49.2895137Z 
2022-11-23T02:21:49.2895286Z OK (skipped=1)
2022-11-23T02:21:49.2895488Z 
2022-11-23T02:21:49.2895651Z Generating XML reports...
2022-11-23T02:21:49.2896353Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123022109.xml
2022-11-23T02:21:49.2897172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2897704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2898360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2898895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2899445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekt0adrl
2022-11-23T02:21:49.2900066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekt0adrl/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2900555Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2900801Z 
2022-11-23T02:21:49.2900950Z Running tests...
2022-11-23T02:21:49.2901431Z ----------------------------------------------------------------------
2022-11-23T02:21:49.2902047Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T02:21:49.2902682Z   test_all_to_all (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58707
2022-11-23T02:21:49.2903306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58708
2022-11-23T02:21:49.2904000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2904536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2905174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2905722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2906477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2906981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2907630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2908172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2908719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7d39m0c
2022-11-23T02:21:49.2909324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7d39m0c/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2909940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpclurv38k
2022-11-23T02:21:49.2910556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpclurv38k/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2911052Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2911539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:49.2912113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:49.2912601Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2913067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:49.2913634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:49.2914433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:49.2915188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:49.2915842Z [1669170076.420647] [2ed9f7ebac38:58707:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2022-11-23T02:21:49.2916443Z [1669170076.431399] [2ed9f7ebac38:58707:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-11-23T02:21:49.2917007Z [1669170076.431399] [2ed9f7ebac38:58707:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-11-23T02:21:49.2917592Z [1669170076.426295] [2ed9f7ebac38:58708:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2022-11-23T02:21:49.2918186Z [1669170076.435750] [2ed9f7ebac38:58708:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-11-23T02:21:49.2918749Z [1669170076.435750] [2ed9f7ebac38:58708:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-11-23T02:21:49.2919175Z ok (4.583s)
2022-11-23T02:21:49.2919363Z 
2022-11-23T02:21:49.2919660Z ----------------------------------------------------------------------
2022-11-23T02:21:49.2920064Z Ran 1 test in 4.583s
2022-11-23T02:21:49.2920264Z 
2022-11-23T02:21:49.2920398Z OK
2022-11-23T02:21:49.2920571Z 
2022-11-23T02:21:49.2920735Z Generating XML reports...
2022-11-23T02:21:49.2921434Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123022112.xml
2022-11-23T02:21:49.2922245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2922780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2923418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2923971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2924521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5whqdtps
2022-11-23T02:21:49.2925209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5whqdtps/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2925935Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2926184Z 
2022-11-23T02:21:49.2926335Z Running tests...
2022-11-23T02:21:49.2926825Z ----------------------------------------------------------------------
2022-11-23T02:21:49.2927426Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T02:21:49.2928088Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58828
2022-11-23T02:21:49.2928718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58829
2022-11-23T02:21:49.2929405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2929918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2930580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2931130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2931789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2932297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2932946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2933573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2934108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9269zh4r
2022-11-23T02:21:49.2934726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9269zh4r/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2935341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxehl7sz
2022-11-23T02:21:49.2935955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxehl7sz/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2936444Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2936928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:49.2937493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:49.2937959Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2938441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:49.2939003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:49.2939748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:49.2940508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:49.2941164Z [1669170084.212260] [2ed9f7ebac38:58828:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2022-11-23T02:21:49.2941760Z [1669170084.224438] [2ed9f7ebac38:58828:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-11-23T02:21:49.2942322Z [1669170084.224438] [2ed9f7ebac38:58828:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-11-23T02:21:49.2942914Z [1669170084.216016] [2ed9f7ebac38:58829:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2022-11-23T02:21:49.2943500Z [1669170084.226992] [2ed9f7ebac38:58829:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-11-23T02:21:49.2944137Z [1669170084.226992] [2ed9f7ebac38:58829:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-11-23T02:21:49.2944561Z ok (4.479s)
2022-11-23T02:21:49.2944729Z 
2022-11-23T02:21:49.2945040Z ----------------------------------------------------------------------
2022-11-23T02:21:49.2945448Z Ran 1 test in 4.479s
2022-11-23T02:21:49.2945652Z 
2022-11-23T02:21:49.2945784Z OK
2022-11-23T02:21:49.2945960Z 
2022-11-23T02:21:49.2946105Z Generating XML reports...
2022-11-23T02:21:49.2946830Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123022120.xml
2022-11-23T02:21:49.2947651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2948184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2948822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2949377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2949923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhefo6t9
2022-11-23T02:21:49.2950527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhefo6t9/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2951039Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2951276Z 
2022-11-23T02:21:49.2951425Z Running tests...
2022-11-23T02:21:49.2951912Z ----------------------------------------------------------------------
2022-11-23T02:21:49.2952569Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T02:21:49.2953228Z   test_allreduce (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58949
2022-11-23T02:21:49.2953851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58950
2022-11-23T02:21:49.2954546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2955058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2955711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2956260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2956900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2957431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2958086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2958630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2959163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ezqx0y6
2022-11-23T02:21:49.2959782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ezqx0y6/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2960392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6yp7ivm2
2022-11-23T02:21:49.2960989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6yp7ivm2/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2961499Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2961985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:49.2962558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:49.2963027Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2963508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:49.2964071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:49.2964860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:49.2965630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:49.2966589Z [1669170092.060359] [2ed9f7ebac38:58950:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2022-11-23T02:21:49.2967192Z [1669170092.070621] [2ed9f7ebac38:58950:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-11-23T02:21:49.2967738Z [1669170092.070621] [2ed9f7ebac38:58950:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-11-23T02:21:49.2968347Z [1669170092.056654] [2ed9f7ebac38:58949:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2022-11-23T02:21:49.2968941Z [1669170092.068738] [2ed9f7ebac38:58949:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-11-23T02:21:49.2969497Z [1669170092.068738] [2ed9f7ebac38:58949:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-11-23T02:21:49.2969903Z ok (4.580s)
2022-11-23T02:21:49.2970091Z 
2022-11-23T02:21:49.2970405Z ----------------------------------------------------------------------
2022-11-23T02:21:49.2970813Z Ran 1 test in 4.580s
2022-11-23T02:21:49.2971013Z 
2022-11-23T02:21:49.2971135Z OK
2022-11-23T02:21:49.2971374Z 
2022-11-23T02:21:49.2971545Z Generating XML reports...
2022-11-23T02:21:49.2972264Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123022128.xml
2022-11-23T02:21:49.2973083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2973600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2974258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2974814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2975361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpex3jmqyo
2022-11-23T02:21:49.2975966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpex3jmqyo/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2976479Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2976718Z 
2022-11-23T02:21:49.2976870Z Running tests...
2022-11-23T02:21:49.2977346Z ----------------------------------------------------------------------
2022-11-23T02:21:49.2977962Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T02:21:49.2978618Z   test_broadcast (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59068
2022-11-23T02:21:49.2979242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59069
2022-11-23T02:21:49.2979911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2980438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2981089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2981645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2982284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2982815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2983552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2984076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.2984626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkuvh24q
2022-11-23T02:21:49.2985249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkuvh24q/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2985861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ecfnzh1
2022-11-23T02:21:49.2986460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ecfnzh1/_remote_module_non_scriptable.py
2022-11-23T02:21:49.2986968Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2987452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:49.2988002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:49.2988494Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.2988979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:49.2989548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:49.2990276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:49.2991047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:49.2991767Z [1669170099.792871] [2ed9f7ebac38:59068:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2022-11-23T02:21:49.2992373Z [1669170099.805337] [2ed9f7ebac38:59068:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-11-23T02:21:49.2992920Z [1669170099.805337] [2ed9f7ebac38:59068:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-11-23T02:21:49.2993526Z [1669170099.792844] [2ed9f7ebac38:59069:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2022-11-23T02:21:49.2994157Z [1669170099.804436] [2ed9f7ebac38:59069:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-11-23T02:21:49.2994714Z [1669170099.804436] [2ed9f7ebac38:59069:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-11-23T02:21:49.2995126Z ok (4.479s)
2022-11-23T02:21:49.2995318Z 
2022-11-23T02:21:49.2995632Z ----------------------------------------------------------------------
2022-11-23T02:21:49.2996039Z Ran 1 test in 4.479s
2022-11-23T02:21:49.2996243Z 
2022-11-23T02:21:49.2996355Z OK
2022-11-23T02:21:49.2996531Z 
2022-11-23T02:21:49.2996692Z Generating XML reports...
2022-11-23T02:21:49.2997415Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123022136.xml
2022-11-23T02:21:49.2998228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.2998742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.2999400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.2999953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.3000489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2946e46
2022-11-23T02:21:49.3001112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2946e46/_remote_module_non_scriptable.py
2022-11-23T02:21:49.3001622Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.3001860Z 
2022-11-23T02:21:49.3002009Z Running tests...
2022-11-23T02:21:49.3002547Z ----------------------------------------------------------------------
2022-11-23T02:21:49.3003164Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2022-11-23T02:21:49.3003810Z   test_reduce (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59187
2022-11-23T02:21:49.3004408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59188
2022-11-23T02:21:49.3005096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.3005632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.3006542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.3007075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.3007735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:49.3008259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:49.3008911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:49.3009440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:49.3009985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1x6w5kg
2022-11-23T02:21:49.3010676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1x6w5kg/_remote_module_non_scriptable.py
2022-11-23T02:21:49.3011281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23wwukl7
2022-11-23T02:21:49.3011901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23wwukl7/_remote_module_non_scriptable.py
2022-11-23T02:21:49.3012408Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.3012898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:49.3013449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:49.3013931Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:49.3014411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:49.3014963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:49.3015707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:49.3016479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:49.3017129Z [1669170107.540498] [2ed9f7ebac38:59188:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2022-11-23T02:21:49.3017714Z [1669170107.550937] [2ed9f7ebac38:59188:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-11-23T02:21:49.3018279Z [1669170107.550937] [2ed9f7ebac38:59188:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-11-23T02:21:49.3018880Z [1669170107.537431] [2ed9f7ebac38:59187:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2022-11-23T02:21:49.3019469Z [1669170107.548926] [2ed9f7ebac38:59187:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-11-23T02:21:49.3020006Z [1669170107.548926] [2ed9f7ebac38:59187:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-11-23T02:21:49.3020444Z ok (4.480s)
2022-11-23T02:21:49.3020635Z 
2022-11-23T02:21:49.3020944Z ----------------------------------------------------------------------
2022-11-23T02:21:49.3021426Z Ran 1 test in 4.480s
2022-11-23T02:21:49.3021609Z 
2022-11-23T02:21:49.3021748Z OK
2022-11-23T02:21:49.3021920Z 
2022-11-23T02:21:49.3022086Z Generating XML reports...
2022-11-23T02:21:49.3022802Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123022144.xml
2022-11-23T02:21:49.3023231Z 
2022-11-23T02:21:49.3023656Z ##[endgroup]
2022-11-23T02:21:49.3024314Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_ucc_arovbp1u)
2022-11-23T02:21:49.3024698Z 
2022-11-23T02:21:49.3025028Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2022-11-23 02:21:49.287152]
2022-11-23T02:21:49.3025813Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:21:49.287501]
2022-11-23T02:22:17.3965552Z 
2022-11-23T02:22:17.3968530Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_clip_grad_norm
2022-11-23T02:22:17.3969605Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_f1v26l9u)
2022-11-23T02:22:17.3970036Z 
2022-11-23T02:22:17.3970182Z Running tests...
2022-11-23T02:22:17.3971068Z ----------------------------------------------------------------------
2022-11-23T02:22:17.3973013Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm
2022-11-23T02:22:17.3973676Z   test_ddp_parity (__main__.TestClipGradNorm)
2022-11-23T02:22:17.3974456Z Tests FSDP with ``FullyShardedDataParallel.clip_grad_norm_()` against ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:22:17.3975059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59306
2022-11-23T02:22:17.3975567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59307
2022-11-23T02:22:17.3976309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:17.3976906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:17.3977903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:17.3978939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:17.3980076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:17.3983359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:17.3984114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:17.3984709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:17.3985250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:17.3985840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:17.3986566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:17.3987345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:17.3987941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:17.3988477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:17.3989048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.3989614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.3990361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.3990901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.3991473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.3992094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.3993198Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.3994599Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.3995404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.3995964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.3997040Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.3998419Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.3999759Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4001065Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4002383Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4003176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4003793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4004346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4004906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4005470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4006374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4006938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4007493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4008023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4008683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4009231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4009780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4010315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4010868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4011415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4011948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4013041Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4013861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4014415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4014949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4015502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4016126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4016695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4017226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4017776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4018333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4018864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4019421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4019968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4021052Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4021861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4022399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4022955Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4023509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4024042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4024590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4025136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4025687Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4026744Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4028129Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4029451Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4030767Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4032080Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4032891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4033431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4034047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4034611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4035680Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4036473Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4037036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4037587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4038141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4038675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4039226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4039781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4040310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4040864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4041413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4041962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4042491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4043560Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4044368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4044982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4045513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4046497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4047048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4047596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4048130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4048678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4049752Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4051075Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4052474Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4053780Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4055103Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:22:17.4055911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4056476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4057033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4057574Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:22:17.4058015Z dist init r=0, world=2
2022-11-23T02:22:17.4058351Z dist init r=1, world=2
2022-11-23T02:22:17.4058653Z ok (22.034s)
2022-11-23T02:22:17.4059012Z   test_non_root (__main__.TestClipGradNorm)
2022-11-23T02:22:17.4059724Z Tests that calling ``clip_grad_norm_()`` on a non-root FSDP instance ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59389
2022-11-23T02:22:17.4060322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59390
2022-11-23T02:22:17.4061020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:17.4061555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:17.4062200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:17.4062711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:17.4063367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:17.4064001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:17.4064674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:17.4065201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:17.4065733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:17.4066309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:17.4067030Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:17.4067801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:17.4068403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:17.4068960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:17.4069375Z dist init r=1, world=2
2022-11-23T02:22:17.4069708Z dist init r=0, world=2
2022-11-23T02:22:17.4070029Z ok (3.811s)
2022-11-23T02:22:17.4070219Z 
2022-11-23T02:22:17.4070511Z ----------------------------------------------------------------------
2022-11-23T02:22:17.4070918Z Ran 2 tests in 25.845s
2022-11-23T02:22:17.4071123Z 
2022-11-23T02:22:17.4071257Z OK
2022-11-23T02:22:17.4071431Z 
2022-11-23T02:22:17.4071598Z Generating XML reports...
2022-11-23T02:22:17.4072320Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20221123022151.xml
2022-11-23T02:22:17.4072737Z 
2022-11-23T02:22:17.4073197Z ##[endgroup]
2022-11-23T02:22:17.4073908Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_f1v26l9u)
2022-11-23T02:22:17.4074324Z 
2022-11-23T02:22:17.4074644Z Running distributed/elastic/multiprocessing/api_test ... [2022-11-23 02:22:17.396732]
2022-11-23T02:22:17.4075443Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:22:17.397008]
2022-11-23T02:22:47.2562043Z 
2022-11-23T02:22:47.2562746Z Expand the folded group to see the log file of distributed/elastic/multiprocessing/api_test
2022-11-23T02:22:47.2564697Z ##[group]PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_wqkgkngv)
2022-11-23T02:22:47.2565169Z 
2022-11-23T02:22:47.2565301Z Running tests...
2022-11-23T02:22:47.2566840Z ----------------------------------------------------------------------
2022-11-23T02:22:47.2567567Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test
2022-11-23T02:22:47.2568166Z   test_get_failures (__main__.RunProcResultsTest) ... ok (1.601s)
2022-11-23T02:22:47.2568624Z   test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s)
2022-11-23T02:22:47.2569087Z   test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.002s)
2022-11-23T02:22:47.2571437Z   test_binary (__main__.StartProcessesListTest) ... hello stderr from 1
2022-11-23T02:22:47.2571900Z hello stderr from 0
2022-11-23T02:22:47.2572240Z hello stdout from 1
2022-11-23T02:22:47.2572559Z hello stdout from 0
2022-11-23T02:22:47.2572850Z ok (0.104s)
2022-11-23T02:22:47.2577269Z   test_binary_exit (__main__.StartProcessesListTest) ... bar stderr from 1
2022-11-23T02:22:47.2577692Z bar stdout from 1
2022-11-23T02:22:47.2578227Z failed (exitcode: 138) local_rank: 0 (pid: 59508) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2022-11-23T02:22:47.2578830Z ok (0.104s)
2022-11-23T02:22:47.2579607Z   test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.002s)
2022-11-23T02:22:47.2580143Z   test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last):
2022-11-23T02:22:47.2580722Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-11-23T02:22:47.2581205Z     raise RuntimeError(f"raised from {rank}")
2022-11-23T02:22:47.2581555Z RuntimeError: raised from 0
2022-11-23T02:22:47.2581884Z bar from 1
2022-11-23T02:22:47.2582373Z failed (exitcode: 1) local_rank: 0 (pid: 59511) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2022-11-23T02:22:47.2582822Z ok (0.104s)
2022-11-23T02:22:47.2583233Z   test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1
2022-11-23T02:22:47.2583678Z [trainer1]:world stderr from 1
2022-11-23T02:22:47.2583999Z [trainer0]:hello stdout from 0
2022-11-23T02:22:47.2584326Z ok (1.106s)
2022-11-23T02:22:47.2585081Z   test_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2585678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2586396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2586925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2587699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2588230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2589122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2589976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2590379Z hello stderr from 0
2022-11-23T02:22:47.2590699Z hello stderr from 1
2022-11-23T02:22:47.2591000Z hello stdout from 0
2022-11-23T02:22:47.2591311Z hello stdout from 1
2022-11-23T02:22:47.2591653Z Closing process 59519 via signal SIGTERM
2022-11-23T02:22:47.2591974Z ok (2.415s)
2022-11-23T02:22:47.2592696Z   test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2593297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2593924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2594471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2595118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2595645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2596265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2596782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2597424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2597962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2598606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2599141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2599785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2600281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2601029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2601561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2601975Z Closing process 59584 via signal SIGTERM
2022-11-23T02:22:47.2602327Z Closing process 59585 via signal SIGTERM
2022-11-23T02:22:47.2602694Z Closing process 59587 via signal SIGTERM
2022-11-23T02:22:47.2603032Z ok (2.779s)
2022-11-23T02:22:47.2603381Z   test_function_raise (__main__.StartProcessesListTest)
2022-11-23T02:22:47.2604150Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2604725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2605349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2606273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2606928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2607446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2608070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2608605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2609177Z failed (exitcode: 1) local_rank: 0 (pid: 59716) of fn: echo2 (start_method: spawn)
2022-11-23T02:22:47.2609615Z Traceback (most recent call last):
2022-11-23T02:22:47.2610226Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-11-23T02:22:47.2610703Z     self._pc.join(-1)
2022-11-23T02:22:47.2611251Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-11-23T02:22:47.2611780Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-11-23T02:22:47.2612289Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-11-23T02:22:47.2612595Z 
2022-11-23T02:22:47.2612856Z -- Process 0 terminated with the following error:
2022-11-23T02:22:47.2613244Z Traceback (most recent call last):
2022-11-23T02:22:47.2613810Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-11-23T02:22:47.2614218Z     fn(i, *args)
2022-11-23T02:22:47.2614784Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2022-11-23T02:22:47.2615242Z     ret = record(fn)(*args_)
2022-11-23T02:22:47.2615832Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2022-11-23T02:22:47.2616313Z     return f(*args, **kwargs)
2022-11-23T02:22:47.2616768Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-11-23T02:22:47.2617210Z     raise RuntimeError(msg)
2022-11-23T02:22:47.2617524Z RuntimeError: hello
2022-11-23T02:22:47.2617717Z 
2022-11-23T02:22:47.2617844Z ok (2.422s)
2022-11-23T02:22:47.2618234Z   test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s)
2022-11-23T02:22:47.2618701Z   test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.001s)
2022-11-23T02:22:47.2619238Z   test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 59782 via signal SIGTERM
2022-11-23T02:22:47.2619676Z ok (0.004s)
2022-11-23T02:22:47.2620336Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-11-23T02:22:47.2620870Z Traceback (most recent call last):
2022-11-23T02:22:47.2621568Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-11-23T02:22:47.2622060Z     self._pc.join(-1)
2022-11-23T02:22:47.2622435Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2022-11-23T02:22:47.2622846Z     return self._mock_call(*args, **kwargs)
2022-11-23T02:22:47.2623262Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2022-11-23T02:22:47.2623667Z     return self._execute_mock_call(*args, **kwargs)
2022-11-23T02:22:47.2624108Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2022-11-23T02:22:47.2624506Z     raise effect
2022-11-23T02:22:47.2624925Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-11-23T02:22:47.2625353Z ok (0.005s)
2022-11-23T02:22:47.2626040Z   test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2626627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2627252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2627785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2628158Z ok (3.263s)
2022-11-23T02:22:47.2628595Z   test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 59816 closing signal SIGTERM
2022-11-23T02:22:47.2629022Z ok (0.004s)
2022-11-23T02:22:47.2629447Z   test_to_map (__main__.StartProcessesListTest) ... ok (0.002s)
2022-11-23T02:22:47.2629927Z   test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s)
2022-11-23T02:22:47.2630701Z   test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2631290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2631926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2632462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2633092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2633600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2634235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2634752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2635119Z hello
2022-11-23T02:22:47.2635390Z world
2022-11-23T02:22:47.2635690Z Closing process 59818 via signal SIGTERM
2022-11-23T02:22:47.2636013Z ok (2.410s)
2022-11-23T02:22:47.2636407Z   test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s)
2022-11-23T02:22:47.2636858Z   test_binary_exit (__main__.StartProcessesTest) ... bar stderr from 1
2022-11-23T02:22:47.2637246Z bar stdout from 1
2022-11-23T02:22:47.2637742Z failed (exitcode: 138) local_rank: 0 (pid: 59883) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2022-11-23T02:22:47.2638205Z ok (0.104s)
2022-11-23T02:22:47.2638584Z   test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.002s)
2022-11-23T02:22:47.2639088Z   test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last):
2022-11-23T02:22:47.2639637Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-11-23T02:22:47.2640099Z     raise RuntimeError(f"raised from {rank}")
2022-11-23T02:22:47.2640461Z RuntimeError: raised from 0
2022-11-23T02:22:47.2640779Z bar from 1
2022-11-23T02:22:47.2641238Z failed (exitcode: 1) local_rank: 0 (pid: 59886) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2022-11-23T02:22:47.2641769Z ok (0.104s)
2022-11-23T02:22:47.2642463Z   test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2643050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2643680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2644213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2644860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2645372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2646561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2647093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2647740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2648252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2648875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2649408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2650154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2650684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2651313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2651850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2652253Z Closing process 59889 via signal SIGTERM
2022-11-23T02:22:47.2652598Z Closing process 59890 via signal SIGTERM
2022-11-23T02:22:47.2652960Z Closing process 59891 via signal SIGTERM
2022-11-23T02:22:47.2653292Z ok (2.777s)
2022-11-23T02:22:47.2653621Z   test_function_raise (__main__.StartProcessesTest)
2022-11-23T02:22:47.2654374Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2654954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2655596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2656116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2656773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2657284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2657918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2658434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2658905Z failed (exitcode: 1) local_rank: 0 (pid: 60020) of fn: echo2 (start_method: spawn)
2022-11-23T02:22:47.2659318Z Traceback (most recent call last):
2022-11-23T02:22:47.2659910Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-11-23T02:22:47.2660412Z     self._pc.join(-1)
2022-11-23T02:22:47.2660949Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-11-23T02:22:47.2661565Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-11-23T02:22:47.2662070Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-11-23T02:22:47.2662378Z 
2022-11-23T02:22:47.2662637Z -- Process 0 terminated with the following error:
2022-11-23T02:22:47.2663021Z Traceback (most recent call last):
2022-11-23T02:22:47.2663566Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-11-23T02:22:47.2663993Z     fn(i, *args)
2022-11-23T02:22:47.2664562Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2022-11-23T02:22:47.2665006Z     ret = record(fn)(*args_)
2022-11-23T02:22:47.2665615Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2022-11-23T02:22:47.2666080Z     return f(*args, **kwargs)
2022-11-23T02:22:47.2666538Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-11-23T02:22:47.2666972Z     raise RuntimeError(msg)
2022-11-23T02:22:47.2667304Z RuntimeError: hello
2022-11-23T02:22:47.2667496Z 
2022-11-23T02:22:47.2667632Z ok (2.322s)
2022-11-23T02:22:47.2667997Z   test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s)
2022-11-23T02:22:47.2668461Z   test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.002s)
2022-11-23T02:22:47.2668972Z   test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 60086 via signal SIGTERM
2022-11-23T02:22:47.2669399Z ok (0.003s)
2022-11-23T02:22:47.2670113Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-11-23T02:22:47.2670649Z Traceback (most recent call last):
2022-11-23T02:22:47.2671255Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-11-23T02:22:47.2671741Z     self._pc.join(-1)
2022-11-23T02:22:47.2672132Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2022-11-23T02:22:47.2672541Z     return self._mock_call(*args, **kwargs)
2022-11-23T02:22:47.2672943Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2022-11-23T02:22:47.2673374Z     return self._execute_mock_call(*args, **kwargs)
2022-11-23T02:22:47.2673817Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2022-11-23T02:22:47.2674206Z     raise effect
2022-11-23T02:22:47.2674624Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-11-23T02:22:47.2675050Z ok (0.004s)
2022-11-23T02:22:47.2675730Z   test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2676292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2676942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2677481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2677855Z ok (3.370s)
2022-11-23T02:22:47.2678272Z   test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 60120 closing signal SIGTERM
2022-11-23T02:22:47.2678700Z ok (0.004s)
2022-11-23T02:22:47.2679048Z   test_to_map (__main__.StartProcessesTest) ... ok (0.001s)
2022-11-23T02:22:47.2679476Z   test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s)
2022-11-23T02:22:47.2680239Z   test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2680815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2681455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2682072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2682715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:47.2683230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:47.2683850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:47.2684380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:47.2684745Z world
2022-11-23T02:22:47.2684995Z hello
2022-11-23T02:22:47.2685311Z Closing process 60122 via signal SIGTERM
2022-11-23T02:22:47.2685646Z ok (2.414s)
2022-11-23T02:22:47.2686349Z   test_from_str_bad_input (__main__.StdTest) ... ok (0.001s)
2022-11-23T02:22:47.2686755Z   test_from_value (__main__.StdTest) ... ok (0.001s)
2022-11-23T02:22:47.2687148Z   test_from_value_map (__main__.StdTest) ... ok (0.001s)
2022-11-23T02:22:47.2687390Z 
2022-11-23T02:22:47.2687702Z ----------------------------------------------------------------------
2022-11-23T02:22:47.2688083Z Ran 38 tests in 27.452s
2022-11-23T02:22:47.2688278Z 
2022-11-23T02:22:47.2688403Z OK
2022-11-23T02:22:47.2688569Z 
2022-11-23T02:22:47.2688727Z Generating XML reports...
2022-11-23T02:22:47.2689426Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20221123022219.xml
2022-11-23T02:22:47.2690503Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20221123022219.xml
2022-11-23T02:22:47.2691451Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20221123022219.xml
2022-11-23T02:22:47.2692324Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20221123022219.xml
2022-11-23T02:22:47.2692726Z 
2022-11-23T02:22:47.2693278Z ##[endgroup]
2022-11-23T02:22:47.2694003Z FINISHED PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_wqkgkngv)
2022-11-23T02:22:47.2694452Z 
2022-11-23T02:22:47.2694757Z Running distributed/_tensor/test_matrix_ops ... [2022-11-23 02:22:47.256436]
2022-11-23T02:22:47.2695506Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:22:47.256739]
2022-11-23T02:23:19.6461637Z 
2022-11-23T02:23:19.6462490Z Expand the folded group to see the log file of distributed/_tensor/test_matrix_ops
2022-11-23T02:23:19.6463534Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_matrix_ops_lsohs0du)
2022-11-23T02:23:19.6463934Z 
2022-11-23T02:23:19.6464078Z Running tests...
2022-11-23T02:23:19.6466435Z ----------------------------------------------------------------------
2022-11-23T02:23:19.6467117Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_matrix_ops
2022-11-23T02:23:19.6467699Z   test_addmm (__main__.DistMatrixOpsTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:23:19.6468265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60222
2022-11-23T02:23:19.6468982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60223
2022-11-23T02:23:19.6469713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6470259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6470980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6471638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6472552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6473080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6473746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6474282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6474796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:19.6475366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:19.6475931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:19.6476471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:19.6477219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6477996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6478462Z ok (5.328s)
2022-11-23T02:23:19.6478960Z   test_addmm_auto_redistribute (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60303
2022-11-23T02:23:19.6479570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60304
2022-11-23T02:23:19.6480359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6480884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6481538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6482088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6482746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6483252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6483951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6484499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6485015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:19.6485569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:19.6486496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:19.6487058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:19.6487845Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6488592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6489068Z ok (3.811s)
2022-11-23T02:23:19.6489555Z   test_baddbmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60388
2022-11-23T02:23:19.6490333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60389
2022-11-23T02:23:19.6491020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6491548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6492198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6492845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6493504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6494027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6494673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6495195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6495713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:19.6496278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:19.6496841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:19.6497381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:19.6498120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6498884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6499338Z ok (5.815s)
2022-11-23T02:23:19.6499819Z   test_bmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60473
2022-11-23T02:23:19.6500470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60474
2022-11-23T02:23:19.6501169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6501677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6502323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6502876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6503512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6504032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6504682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6505219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6505721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:19.6506285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:19.6506844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:19.6507411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:19.6508126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6508888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6509358Z ok (4.012s)
2022-11-23T02:23:19.6509821Z   test_mm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60558
2022-11-23T02:23:19.6510402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60559
2022-11-23T02:23:19.6511092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6511617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6512246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6512858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6513511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6514015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6514662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6515203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6515719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:19.6516267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:19.6516825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:19.6517387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:19.6518117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6518868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6519337Z ok (3.911s)
2022-11-23T02:23:19.6519814Z   test_t (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60643
2022-11-23T02:23:19.6520427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60644
2022-11-23T02:23:19.6521126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6521646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6522305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6522835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6523485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6524003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6524651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6525181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6525982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:19.6526560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:19.6527104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:19.6527669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:19.6528408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6529163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6544115Z ok (3.309s)
2022-11-23T02:23:19.6544648Z   test_t_partial (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60724
2022-11-23T02:23:19.6545240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60725
2022-11-23T02:23:19.6545950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6546480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6547300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6547837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6548498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:19.6549023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:19.6549672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:19.6550204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:19.6550721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:19.6551289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:19.6551860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:19.6552398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:19.6553136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6553905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:19.6554365Z ok (3.810s)
2022-11-23T02:23:19.6554555Z 
2022-11-23T02:23:19.6554948Z ----------------------------------------------------------------------
2022-11-23T02:23:19.6555371Z Ran 7 tests in 29.997s
2022-11-23T02:23:19.6555575Z 
2022-11-23T02:23:19.6555706Z OK
2022-11-23T02:23:19.6555860Z 
2022-11-23T02:23:19.6556025Z Generating XML reports...
2022-11-23T02:23:19.6556703Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_matrix_ops/TEST-DistMatrixOpsTest-20221123022249.xml
2022-11-23T02:23:19.6557097Z 
2022-11-23T02:23:19.6557636Z ##[endgroup]
2022-11-23T02:23:19.6558296Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_matrix_ops_lsohs0du)
2022-11-23T02:23:19.6558692Z 
2022-11-23T02:23:19.6559009Z Running distributed/fsdp/test_fsdp_exec_order ... [2022-11-23 02:23:19.646215]
2022-11-23T02:23:19.6559779Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:23:19.646490]
2022-11-23T02:23:53.6690155Z 
2022-11-23T02:23:53.6691070Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_exec_order
2022-11-23T02:23:53.6692417Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_wz_u69uk)
2022-11-23T02:23:53.6692855Z 
2022-11-23T02:23:53.6697633Z Running tests...
2022-11-23T02:23:53.6698960Z ----------------------------------------------------------------------
2022-11-23T02:23:53.6699840Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order
2022-11-23T02:23:53.6700447Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-11-23T02:23:53.6701165Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:23:53.6701722Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60840
2022-11-23T02:23:53.6702249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60841
2022-11-23T02:23:53.6703410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6704162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6705673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6706659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6707762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6708290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6709344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6710021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6711028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:53.6712048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:53.6713385Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6714710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6715690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:53.6716449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:53.6717951Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6718894Z   warnings.warn(
2022-11-23T02:23:53.6720129Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6720982Z   warnings.warn(
2022-11-23T02:23:53.6721291Z dist init r=1, world=2
2022-11-23T02:23:53.6721583Z dist init r=0, world=2
2022-11-23T02:23:53.6721875Z ok (5.331s)
2022-11-23T02:23:53.6722331Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-11-23T02:23:53.6723108Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60919
2022-11-23T02:23:53.6723730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60920
2022-11-23T02:23:53.6724422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6724942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6725583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6726469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6727135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6727653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6728283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6728818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6729458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:53.6730097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:53.6730834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6731580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6732184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:53.6732732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:53.6734080Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6734933Z   warnings.warn(
2022-11-23T02:23:53.6736238Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6737106Z   warnings.warn(
2022-11-23T02:23:53.6737429Z dist init r=1, world=2
2022-11-23T02:23:53.6737753Z dist init r=0, world=2
2022-11-23T02:23:53.6738057Z ok (3.712s)
2022-11-23T02:23:53.6738548Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-11-23T02:23:53.6739375Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60998
2022-11-23T02:23:53.6739990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60999
2022-11-23T02:23:53.6740658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6741186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6741834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6742365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6743013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6743536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6744181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6744701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6745225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:53.6745796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:53.6746511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6747271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6747935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:53.6748479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:53.6749813Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6750672Z   warnings.warn(
2022-11-23T02:23:53.6751901Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6752745Z   warnings.warn(
2022-11-23T02:23:53.6753066Z dist init r=0, world=2
2022-11-23T02:23:53.6753369Z dist init r=1, world=2
2022-11-23T02:23:53.6753676Z ok (3.711s)
2022-11-23T02:23:53.6754167Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-11-23T02:23:53.6755021Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61081
2022-11-23T02:23:53.6755650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61082
2022-11-23T02:23:53.6756334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6756858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6757490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6758028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6758682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6759195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6759831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6760374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6760899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:53.6761451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:53.6762192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6762952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6763541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:53.6764067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:53.6765399Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6766691Z   warnings.warn(
2022-11-23T02:23:53.6767935Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6768771Z   warnings.warn(
2022-11-23T02:23:53.6769081Z dist init r=1, world=2
2022-11-23T02:23:53.6769404Z dist init r=0, world=2
2022-11-23T02:23:53.6769715Z ok (3.812s)
2022-11-23T02:23:53.6770192Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-11-23T02:23:53.6771018Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61164
2022-11-23T02:23:53.6771635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61165
2022-11-23T02:23:53.6772315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6772820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6773465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6774091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6774759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6775258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6775901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6776440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6776947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:53.6777677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6778281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:53.6779003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6779591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:53.6780129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:53.6781472Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6782333Z   warnings.warn(
2022-11-23T02:23:53.6783564Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6784395Z   warnings.warn(
2022-11-23T02:23:53.6784853Z dist init r=0, world=2
2022-11-23T02:23:53.6785180Z dist init r=1, world=2
2022-11-23T02:23:53.6785470Z ok (3.811s)
2022-11-23T02:23:53.6785967Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-11-23T02:23:53.6786800Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61247
2022-11-23T02:23:53.6787417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61248
2022-11-23T02:23:53.6788087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6788609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6789254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6789798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6790429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6790948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6791594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6792112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6792694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:53.6793275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:53.6794005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6794751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6795351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:53.6795899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:53.6797242Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6798097Z   warnings.warn(
2022-11-23T02:23:53.6799305Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6800161Z   warnings.warn(
2022-11-23T02:23:53.6800477Z dist init r=0, world=2
2022-11-23T02:23:53.6800802Z dist init r=1, world=2
2022-11-23T02:23:53.6801094Z ok (3.711s)
2022-11-23T02:23:53.6801651Z   test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61330
2022-11-23T02:23:53.6802290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61331
2022-11-23T02:23:53.6802959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6803547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6804202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6804740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6805372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6806176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6806828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6807352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6807878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:53.6808459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:53.6809185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6809932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6810526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:53.6811070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:53.6812488Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6813365Z   warnings.warn(
2022-11-23T02:23:53.6814576Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6815427Z   warnings.warn(
2022-11-23T02:23:53.6815742Z dist init r=1, world=2
2022-11-23T02:23:53.6816065Z dist init r=0, world=2
2022-11-23T02:23:53.6816364Z ok (3.811s)
2022-11-23T02:23:53.6816917Z   test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61413
2022-11-23T02:23:53.6817565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61414
2022-11-23T02:23:53.6818233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6818757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6819407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6819948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6820591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:53.6821114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:53.6821756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:53.6822280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:53.6822917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:23:53.6823485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:23:53.6824216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6824960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:23:53.6825558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:53.6826103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:53.6827443Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6828299Z   warnings.warn(
2022-11-23T02:23:53.6829565Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:23:53.6830413Z   warnings.warn(
2022-11-23T02:23:53.6830734Z dist init r=0, world=2
2022-11-23T02:23:53.6831057Z dist init r=1, world=2
2022-11-23T02:23:53.6831350Z ok (3.811s)
2022-11-23T02:23:53.6831542Z 
2022-11-23T02:23:53.6831848Z ----------------------------------------------------------------------
2022-11-23T02:23:53.6832249Z Ran 8 tests in 31.710s
2022-11-23T02:23:53.6832445Z 
2022-11-23T02:23:53.6832559Z OK
2022-11-23T02:23:53.6832727Z 
2022-11-23T02:23:53.6832886Z Generating XML reports...
2022-11-23T02:23:53.6833564Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20221123022321.xml
2022-11-23T02:23:53.6833958Z 
2022-11-23T02:23:53.6834404Z ##[endgroup]
2022-11-23T02:23:53.6835094Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_wz_u69uk)
2022-11-23T02:23:53.6835496Z 
2022-11-23T02:23:53.6835800Z Running distributed/test_c10d_spawn_gloo ... [2022-11-23 02:23:53.669133]
2022-11-23T02:23:53.6836574Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:23:53.669430]
2022-11-23T02:25:07.6325823Z 
2022-11-23T02:25:07.6328705Z Expand the folded group to see the log file of distributed/test_c10d_spawn_gloo
2022-11-23T02:25:07.6329755Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_3idpvnqn)
2022-11-23T02:25:07.6330432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdr6y7q_h
2022-11-23T02:25:07.6331069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdr6y7q_h/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6331600Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6332409Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelSingleProcessTest testMethod=test_cpu>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]>
2022-11-23T02:25:07.6334186Z test_cpu (__main__.DistributedDataParallelSingleProcessTest)
2022-11-23T02:25:07.6334954Z test_cuda (__main__.DistributedDataParallelSingleProcessTest)
2022-11-23T02:25:07.6335706Z test_rnn (__main__.DistributedDataParallelSingleProcessTest)
2022-11-23T02:25:07.6336322Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:25:07.6347862Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:25:07.6349127Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsGloo testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]>
2022-11-23T02:25:07.6350476Z test_all_gather (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:25:07.6350951Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:25:07.6351422Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:25:07.6351898Z test_allreduce (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:25:07.6352342Z test_broadcast (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:25:07.6352798Z test_gather (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:25:07.6353255Z test_reduce (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:25:07.6353688Z test_scatter (__main__.TestDistributedNNFunctionsGloo)
2022-11-23T02:25:07.6354686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6355241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6355885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6356446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6356993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnid8vhxu
2022-11-23T02:25:07.6357674Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnid8vhxu/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6358173Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6358408Z 
2022-11-23T02:25:07.6358538Z Running tests...
2022-11-23T02:25:07.6359019Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6359639Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:25:07.6360342Z   test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:07.6360869Z ok (0.023s)
2022-11-23T02:25:07.6361319Z 
2022-11-23T02:25:07.6361646Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6362043Z Ran 1 test in 0.023s
2022-11-23T02:25:07.6362236Z 
2022-11-23T02:25:07.6362339Z OK
2022-11-23T02:25:07.6362509Z 
2022-11-23T02:25:07.6362670Z Generating XML reports...
2022-11-23T02:25:07.6363430Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123022359.xml
2022-11-23T02:25:07.6364284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6364787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6365435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6366448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6366972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1r_ymloy
2022-11-23T02:25:07.6367706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1r_ymloy/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6368194Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6368422Z 
2022-11-23T02:25:07.6368566Z Running tests...
2022-11-23T02:25:07.6369032Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6369641Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:25:07.6370334Z   test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:07.6370864Z ok (0.446s)
2022-11-23T02:25:07.6371047Z 
2022-11-23T02:25:07.6371343Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6371736Z Ran 1 test in 0.447s
2022-11-23T02:25:07.6371923Z 
2022-11-23T02:25:07.6372049Z OK
2022-11-23T02:25:07.6372196Z 
2022-11-23T02:25:07.6372347Z Generating XML reports...
2022-11-23T02:25:07.6373100Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123022403.xml
2022-11-23T02:25:07.6373956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6374457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6375103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6375712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6376252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8fatvzp
2022-11-23T02:25:07.6376844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8fatvzp/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6377337Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6377566Z 
2022-11-23T02:25:07.6377711Z Running tests...
2022-11-23T02:25:07.6378159Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6378766Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:25:07.6379494Z   test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:25:07.6380030Z ok (1.146s)
2022-11-23T02:25:07.6380205Z 
2022-11-23T02:25:07.6380485Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6380870Z Ran 1 test in 1.146s
2022-11-23T02:25:07.6381058Z 
2022-11-23T02:25:07.6381183Z OK
2022-11-23T02:25:07.6381345Z 
2022-11-23T02:25:07.6381479Z Generating XML reports...
2022-11-23T02:25:07.6382226Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123022407.xml
2022-11-23T02:25:07.6383075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6383590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6384217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6384750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6385280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqjbj4ja
2022-11-23T02:25:07.6385892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqjbj4ja/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6386369Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6386596Z 
2022-11-23T02:25:07.6386741Z Running tests...
2022-11-23T02:25:07.6387205Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6387864Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:25:07.6388508Z   test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61682
2022-11-23T02:25:07.6389117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61683
2022-11-23T02:25:07.6389795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6390296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6390944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6391477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6392107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6392622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6393256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6393786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6394295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzy15nsed
2022-11-23T02:25:07.6394905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzy15nsed/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6395560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu88ahqu6
2022-11-23T02:25:07.6396170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu88ahqu6/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6396645Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6397109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:07.6397566Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6398012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:07.6398566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:07.6399127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:07.6399857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6400600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6401061Z ok (4.075s)
2022-11-23T02:25:07.6401236Z 
2022-11-23T02:25:07.6401534Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6401904Z Ran 1 test in 4.075s
2022-11-23T02:25:07.6402093Z 
2022-11-23T02:25:07.6402223Z OK
2022-11-23T02:25:07.6402383Z 
2022-11-23T02:25:07.6402536Z Generating XML reports...
2022-11-23T02:25:07.6403238Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022411.xml
2022-11-23T02:25:07.6404032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6404549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6405191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6406117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6406663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8ev58ru
2022-11-23T02:25:07.6407268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8ev58ru/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6407858Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6408068Z 
2022-11-23T02:25:07.6408216Z Running tests...
2022-11-23T02:25:07.6408686Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6409416Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:25:07.6410034Z   test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61798
2022-11-23T02:25:07.6410644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61799
2022-11-23T02:25:07.6411318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6411835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6412457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6412991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6413636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6414145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6414768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6415296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6415894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23_9exzp
2022-11-23T02:25:07.6416494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23_9exzp/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6417088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyvfy_yf0
2022-11-23T02:25:07.6417683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyvfy_yf0/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6418178Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6418628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:07.6419083Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6419549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:07.6420086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:07.6420646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:07.6421379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6422135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6422581Z ok (4.178s)
2022-11-23T02:25:07.6422757Z 
2022-11-23T02:25:07.6423053Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6423444Z Ran 1 test in 4.178s
2022-11-23T02:25:07.6423634Z 
2022-11-23T02:25:07.6423737Z OK
2022-11-23T02:25:07.6423898Z 
2022-11-23T02:25:07.6424049Z Generating XML reports...
2022-11-23T02:25:07.6424748Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022418.xml
2022-11-23T02:25:07.6425560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6426062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6426700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6427230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6427821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprvq2eito
2022-11-23T02:25:07.6428414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprvq2eito/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6428908Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6429134Z 
2022-11-23T02:25:07.6429276Z Running tests...
2022-11-23T02:25:07.6429727Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6430329Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:25:07.6430981Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61914
2022-11-23T02:25:07.6431594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61915
2022-11-23T02:25:07.6432256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6432777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6433415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6433933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6434576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6435084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6435774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6436296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6436825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvrvsh2mv
2022-11-23T02:25:07.6437431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvrvsh2mv/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6438019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwf4d5gos
2022-11-23T02:25:07.6438619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwf4d5gos/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6439107Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6439573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:07.6440008Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6440470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:07.6441030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:07.6441573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:07.6442305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6443066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6443528Z ok (4.079s)
2022-11-23T02:25:07.6443706Z 
2022-11-23T02:25:07.6443986Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6444373Z Ran 1 test in 4.079s
2022-11-23T02:25:07.6444567Z 
2022-11-23T02:25:07.6444690Z OK
2022-11-23T02:25:07.6444853Z 
2022-11-23T02:25:07.6444988Z Generating XML reports...
2022-11-23T02:25:07.6446054Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022426.xml
2022-11-23T02:25:07.6447080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6447599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6448222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6448863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6449394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyicel6kq
2022-11-23T02:25:07.6450001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyicel6kq/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6450470Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6450698Z 
2022-11-23T02:25:07.6450842Z Running tests...
2022-11-23T02:25:07.6451319Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6451908Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:25:07.6452547Z   test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62030
2022-11-23T02:25:07.6453156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62031
2022-11-23T02:25:07.6453830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6454332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6454975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6455513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6456211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6456736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6457374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6457904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6458420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd6prou5j
2022-11-23T02:25:07.6459037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd6prou5j/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6459643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7_nic7b
2022-11-23T02:25:07.6460241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7_nic7b/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6460714Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6461189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:07.6461641Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6462093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:07.6462645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:07.6463211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:07.6463936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6464674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6465140Z ok (4.082s)
2022-11-23T02:25:07.6465315Z 
2022-11-23T02:25:07.6465611Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6465988Z Ran 1 test in 4.082s
2022-11-23T02:25:07.6466183Z 
2022-11-23T02:25:07.6466310Z OK
2022-11-23T02:25:07.6466471Z 
2022-11-23T02:25:07.6466621Z Generating XML reports...
2022-11-23T02:25:07.6467325Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022433.xml
2022-11-23T02:25:07.6468185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6468699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6469337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6469852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6470383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijmk_dmg
2022-11-23T02:25:07.6470990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijmk_dmg/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6471480Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6471689Z 
2022-11-23T02:25:07.6471832Z Running tests...
2022-11-23T02:25:07.6472294Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6472894Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:25:07.6473523Z   test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62146
2022-11-23T02:25:07.6474135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62147
2022-11-23T02:25:07.6474806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6475319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6476015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6476554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6477199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6477707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6478334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6478861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6479432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8ixqwsx
2022-11-23T02:25:07.6480025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8ixqwsx/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6480624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwev0eawh
2022-11-23T02:25:07.6481227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwev0eawh/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6481721Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6482172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:07.6482624Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6483093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:07.6483630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:07.6484189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:07.6484913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6486024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6486750Z ok (4.074s)
2022-11-23T02:25:07.6486930Z 
2022-11-23T02:25:07.6487237Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6487631Z Ran 1 test in 4.074s
2022-11-23T02:25:07.6487820Z 
2022-11-23T02:25:07.6487921Z OK
2022-11-23T02:25:07.6488079Z 
2022-11-23T02:25:07.6488338Z Generating XML reports...
2022-11-23T02:25:07.6489050Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022440.xml
2022-11-23T02:25:07.6489858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6490359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6491001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6491541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6492052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphbdjm_nr
2022-11-23T02:25:07.6492663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphbdjm_nr/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6493155Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6493386Z 
2022-11-23T02:25:07.6493530Z Running tests...
2022-11-23T02:25:07.6493984Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6494587Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:25:07.6495227Z   test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62262
2022-11-23T02:25:07.6495818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62263
2022-11-23T02:25:07.6496571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6497104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6497738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6498236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6498886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6499424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6500083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6500608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6501144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfe38mqbz
2022-11-23T02:25:07.6501758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfe38mqbz/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6502347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_1kpfkhk
2022-11-23T02:25:07.6502949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_1kpfkhk/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6503449Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6503840Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6504295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:07.6504832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:07.6505392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:07.6505940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:07.6506677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6507438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6507903Z ok (4.076s)
2022-11-23T02:25:07.6508068Z 
2022-11-23T02:25:07.6508432Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6508829Z Ran 1 test in 4.076s
2022-11-23T02:25:07.6509027Z 
2022-11-23T02:25:07.6509154Z OK
2022-11-23T02:25:07.6509323Z 
2022-11-23T02:25:07.6509465Z Generating XML reports...
2022-11-23T02:25:07.6510185Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022448.xml
2022-11-23T02:25:07.6510997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6511519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6512155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6512702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6513240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqkpha3f
2022-11-23T02:25:07.6513843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqkpha3f/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6514345Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6514576Z 
2022-11-23T02:25:07.6514720Z Running tests...
2022-11-23T02:25:07.6515196Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6515792Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:25:07.6516480Z   test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62378
2022-11-23T02:25:07.6517093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62379
2022-11-23T02:25:07.6517775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6518276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6518929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6519464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6520095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6520612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6521258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6521795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6522313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjjeviq8r
2022-11-23T02:25:07.6522924Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjjeviq8r/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6523529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp113fjqxt
2022-11-23T02:25:07.6524117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp113fjqxt/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6524614Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6525090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:07.6525550Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6526192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:07.6526760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:07.6527326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:07.6528049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6528909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6529372Z ok (4.078s)
2022-11-23T02:25:07.6529554Z 
2022-11-23T02:25:07.6529857Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6530238Z Ran 1 test in 4.078s
2022-11-23T02:25:07.6530434Z 
2022-11-23T02:25:07.6530562Z OK
2022-11-23T02:25:07.6530730Z 
2022-11-23T02:25:07.6530889Z Generating XML reports...
2022-11-23T02:25:07.6531585Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022455.xml
2022-11-23T02:25:07.6532409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6532929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6533572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6534102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6534640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5a6xp_w
2022-11-23T02:25:07.6535256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5a6xp_w/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6535757Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6535973Z 
2022-11-23T02:25:07.6536106Z Running tests...
2022-11-23T02:25:07.6536582Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6537261Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2022-11-23T02:25:07.6537899Z   test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62494
2022-11-23T02:25:07.6538507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62495
2022-11-23T02:25:07.6539191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6539712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6540333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6540865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6541512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.6542016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.6542662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.6543193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.6543736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq10uudf7
2022-11-23T02:25:07.6544329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq10uudf7/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6544931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rz62ac1
2022-11-23T02:25:07.6545532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rz62ac1/_remote_module_non_scriptable.py
2022-11-23T02:25:07.6546015Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6546489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:07.6546953Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.6547425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:07.6547968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:07.6548530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:07.6549331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6550091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.6550542Z ok (4.082s)
2022-11-23T02:25:07.6550723Z 
2022-11-23T02:25:07.6551023Z ----------------------------------------------------------------------
2022-11-23T02:25:07.6551408Z Ran 1 test in 4.082s
2022-11-23T02:25:07.6551604Z 
2022-11-23T02:25:07.6551716Z OK
2022-11-23T02:25:07.6551888Z 
2022-11-23T02:25:07.6552047Z Generating XML reports...
2022-11-23T02:25:07.6552759Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022502.xml
2022-11-23T02:25:07.6553184Z 
2022-11-23T02:25:07.6553630Z ##[endgroup]
2022-11-23T02:25:07.6554293Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_3idpvnqn)
2022-11-23T02:25:07.6554676Z 
2022-11-23T02:25:07.6554977Z Running distributed/test_c10d_spawn_nccl ... [2022-11-23 02:25:07.632943]
2022-11-23T02:25:07.6555752Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:25:07.633270]
2022-11-23T02:26:17.9502644Z 
2022-11-23T02:26:17.9503462Z Expand the folded group to see the log file of distributed/test_c10d_spawn_nccl
2022-11-23T02:26:17.9504984Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_hjtqpg_e)
2022-11-23T02:26:17.9507087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4l3ywfe
2022-11-23T02:26:17.9507738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4l3ywfe/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9508289Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9508692Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:26:17.9509077Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:26:17.9512849Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather_base>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter_non_contiguous>]>
2022-11-23T02:26:17.9514380Z test_all_gather (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:26:17.9514966Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:26:17.9515588Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:26:17.9516066Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:26:17.9516535Z test_allreduce (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:26:17.9517003Z test_broadcast (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:26:17.9517447Z test_reduce (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:26:17.9517905Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:26:17.9518414Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl)
2022-11-23T02:26:17.9519395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9520251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9521341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9521884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9522396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1lxh6bc6
2022-11-23T02:26:17.9523011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1lxh6bc6/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9523509Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9523754Z 
2022-11-23T02:26:17.9523942Z Running tests...
2022-11-23T02:26:17.9524409Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9525251Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:26:17.9526359Z   test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62644
2022-11-23T02:26:17.9526987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62645
2022-11-23T02:26:17.9527665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9528207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9528851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9529399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9530165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9530691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9531332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9531869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9532395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1g7t76a9
2022-11-23T02:26:17.9533008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1g7t76a9/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9533604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4e8t284t
2022-11-23T02:26:17.9534190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4e8t284t/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9534688Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9535173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:26:17.9535733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:26:17.9536194Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9536666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:26:17.9537221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:26:17.9537939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9538703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9539171Z ok (4.176s)
2022-11-23T02:26:17.9539357Z 
2022-11-23T02:26:17.9539660Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9540042Z Ran 1 test in 4.176s
2022-11-23T02:26:17.9540245Z 
2022-11-23T02:26:17.9540374Z OK
2022-11-23T02:26:17.9540545Z 
2022-11-23T02:26:17.9540704Z Generating XML reports...
2022-11-23T02:26:17.9541404Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022513.xml
2022-11-23T02:26:17.9542322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9542841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9543484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9544004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9544534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgabq271y
2022-11-23T02:26:17.9545148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgabq271y/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9545634Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9545870Z 
2022-11-23T02:26:17.9546013Z Running tests...
2022-11-23T02:26:17.9546486Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9547095Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:26:17.9547741Z   test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62763
2022-11-23T02:26:17.9548360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62764
2022-11-23T02:26:17.9549036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9549558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9550248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9550799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9551448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9551952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9552602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9553139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9553672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgwslvyh
2022-11-23T02:26:17.9554268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgwslvyh/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9554871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_c8dovo
2022-11-23T02:26:17.9555478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_c8dovo/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9555962Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9556434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:26:17.9556990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:26:17.9557466Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9557925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:26:17.9558480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:26:17.9559210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9560126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9561105Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:26:17.9561734Z   warnings.warn(
2022-11-23T02:26:17.9562633Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:26:17.9563249Z   warnings.warn(
2022-11-23T02:26:17.9564071Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2923: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-11-23T02:26:17.9564696Z   warnings.warn(
2022-11-23T02:26:17.9565529Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2923: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2022-11-23T02:26:17.9566424Z   warnings.warn(
2022-11-23T02:26:17.9566707Z ok (4.080s)
2022-11-23T02:26:17.9566890Z 
2022-11-23T02:26:17.9567206Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9567606Z Ran 1 test in 4.081s
2022-11-23T02:26:17.9567806Z 
2022-11-23T02:26:17.9567917Z OK
2022-11-23T02:26:17.9568088Z 
2022-11-23T02:26:17.9568247Z Generating XML reports...
2022-11-23T02:26:17.9568962Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022521.xml
2022-11-23T02:26:17.9569779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9570370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9571037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9571576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9572110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkq2_kkon
2022-11-23T02:26:17.9572708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkq2_kkon/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9573208Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9573439Z 
2022-11-23T02:26:17.9573584Z Running tests...
2022-11-23T02:26:17.9574042Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9574696Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:26:17.9575336Z   test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62882
2022-11-23T02:26:17.9575945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62883
2022-11-23T02:26:17.9576611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9577135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9577788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9578311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9578962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9579481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9580122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9580648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9581181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvt4lcrvb
2022-11-23T02:26:17.9581790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvt4lcrvb/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9582460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpla34fj8m
2022-11-23T02:26:17.9583067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpla34fj8m/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9583564Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9584035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:26:17.9584577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:26:17.9585050Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9585526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:26:17.9586070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:26:17.9586803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9587571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9588033Z ok (4.079s)
2022-11-23T02:26:17.9588219Z 
2022-11-23T02:26:17.9588503Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9588902Z Ran 1 test in 4.079s
2022-11-23T02:26:17.9589101Z 
2022-11-23T02:26:17.9589228Z OK
2022-11-23T02:26:17.9589396Z 
2022-11-23T02:26:17.9589554Z Generating XML reports...
2022-11-23T02:26:17.9590312Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022528.xml
2022-11-23T02:26:17.9591144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9591663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9592294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9592842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9593379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphj2yatda
2022-11-23T02:26:17.9593990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphj2yatda/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9594473Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9594705Z 
2022-11-23T02:26:17.9594850Z Running tests...
2022-11-23T02:26:17.9595322Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9595924Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:26:17.9596579Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63003
2022-11-23T02:26:17.9597194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63004
2022-11-23T02:26:17.9597876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9598378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9599024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9599563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9600196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9600717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9601359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9601893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9602479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_2bzexy
2022-11-23T02:26:17.9603075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjj_wlqo
2022-11-23T02:26:17.9603676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_2bzexy/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9604287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjj_wlqo/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9604768Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9605161Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9605637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:26:17.9606467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:26:17.9607020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:26:17.9607582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:26:17.9608318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9609067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9609530Z ok (4.175s)
2022-11-23T02:26:17.9609714Z 
2022-11-23T02:26:17.9610017Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9610400Z Ran 1 test in 4.175s
2022-11-23T02:26:17.9610596Z 
2022-11-23T02:26:17.9610806Z OK
2022-11-23T02:26:17.9610982Z 
2022-11-23T02:26:17.9611141Z Generating XML reports...
2022-11-23T02:26:17.9611860Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022536.xml
2022-11-23T02:26:17.9612652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9613181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9613822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9614350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9614884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dywoy2q
2022-11-23T02:26:17.9615492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dywoy2q/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9615991Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9616210Z 
2022-11-23T02:26:17.9616356Z Running tests...
2022-11-23T02:26:17.9616830Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9617437Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:26:17.9618075Z   test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63124
2022-11-23T02:26:17.9618690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63125
2022-11-23T02:26:17.9619369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9619893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9620525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9621070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9621720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9622234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9622970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9623506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9624036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_sjen57m
2022-11-23T02:26:17.9624628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_sjen57m/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9625231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdwvieod
2022-11-23T02:26:17.9625845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdwvieod/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9626345Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9626805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:26:17.9627363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:26:17.9627843Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9628299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:26:17.9628854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:26:17.9629583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9630345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9630854Z ok (4.078s)
2022-11-23T02:26:17.9631044Z 
2022-11-23T02:26:17.9631346Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9631742Z Ran 1 test in 4.079s
2022-11-23T02:26:17.9631939Z 
2022-11-23T02:26:17.9632049Z OK
2022-11-23T02:26:17.9632216Z 
2022-11-23T02:26:17.9632376Z Generating XML reports...
2022-11-23T02:26:17.9633098Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022543.xml
2022-11-23T02:26:17.9633908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9634410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9635052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9635591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9636115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpboj3p3ra
2022-11-23T02:26:17.9636725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpboj3p3ra/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9637225Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9637457Z 
2022-11-23T02:26:17.9637605Z Running tests...
2022-11-23T02:26:17.9638067Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9638676Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:26:17.9639319Z   test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63243
2022-11-23T02:26:17.9639920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63244
2022-11-23T02:26:17.9640596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9641118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9641760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9642293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9643004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9643522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9644162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9644683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9645217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmy_w33r2
2022-11-23T02:26:17.9646145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmy_w33r2/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9646733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph1xw7p4k
2022-11-23T02:26:17.9647338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph1xw7p4k/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9647839Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9648317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:26:17.9648863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:26:17.9649337Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9649809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:26:17.9650352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:26:17.9651166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9651944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9652406Z ok (4.177s)
2022-11-23T02:26:17.9652571Z 
2022-11-23T02:26:17.9652874Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9653277Z Ran 1 test in 4.177s
2022-11-23T02:26:17.9653474Z 
2022-11-23T02:26:17.9653603Z OK
2022-11-23T02:26:17.9653771Z 
2022-11-23T02:26:17.9653912Z Generating XML reports...
2022-11-23T02:26:17.9654623Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022550.xml
2022-11-23T02:26:17.9655433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9655954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9656588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9657138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9657672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrqpblpt
2022-11-23T02:26:17.9658289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrqpblpt/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9658774Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9659007Z 
2022-11-23T02:26:17.9659151Z Running tests...
2022-11-23T02:26:17.9659623Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9660215Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:26:17.9660855Z   test_reduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63362
2022-11-23T02:26:17.9661469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63363
2022-11-23T02:26:17.9662144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9662655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9663381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9663918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9664544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9665063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9665705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9666242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9666762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytsxb38a
2022-11-23T02:26:17.9667377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytsxb38a/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9667980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppm2zpfce
2022-11-23T02:26:17.9668579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppm2zpfce/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9669079Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9669471Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9669948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:26:17.9670491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:26:17.9671097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:26:17.9671662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:26:17.9672375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9673140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9673613Z ok (4.178s)
2022-11-23T02:26:17.9673797Z 
2022-11-23T02:26:17.9674098Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9674527Z Ran 1 test in 4.178s
2022-11-23T02:26:17.9674728Z 
2022-11-23T02:26:17.9674856Z OK
2022-11-23T02:26:17.9675025Z 
2022-11-23T02:26:17.9675185Z Generating XML reports...
2022-11-23T02:26:17.9675883Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022558.xml
2022-11-23T02:26:17.9676703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9677227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9677873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9678403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9678937Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedg5go2a
2022-11-23T02:26:17.9679545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedg5go2a/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9680044Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9680261Z 
2022-11-23T02:26:17.9680406Z Running tests...
2022-11-23T02:26:17.9680880Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9681490Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:26:17.9682129Z   test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63481
2022-11-23T02:26:17.9682747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63482
2022-11-23T02:26:17.9683496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9684013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9684638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9685178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9686165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9686676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9687319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9687853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9688387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplctnifxi
2022-11-23T02:26:17.9688987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplctnifxi/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9689593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpisqzuxnp
2022-11-23T02:26:17.9690199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpisqzuxnp/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9690702Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9691165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:26:17.9691800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:26:17.9692283Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9692744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:26:17.9693296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:26:17.9694034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9694799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9695248Z ok (4.177s)
2022-11-23T02:26:17.9695433Z 
2022-11-23T02:26:17.9695734Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9696130Z Ran 1 test in 4.177s
2022-11-23T02:26:17.9696327Z 
2022-11-23T02:26:17.9696437Z OK
2022-11-23T02:26:17.9696604Z 
2022-11-23T02:26:17.9696767Z Generating XML reports...
2022-11-23T02:26:17.9697481Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022605.xml
2022-11-23T02:26:17.9698299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9698814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9699459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9700001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9700518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyxe8djj
2022-11-23T02:26:17.9701125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyxe8djj/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9701626Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9701862Z 
2022-11-23T02:26:17.9702007Z Running tests...
2022-11-23T02:26:17.9702467Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9703079Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2022-11-23T02:26:17.9703756Z   test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63600
2022-11-23T02:26:17.9704461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63601
2022-11-23T02:26:17.9705143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9705662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9706306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9706837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9707486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:17.9708000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:17.9708645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:17.9709176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:17.9709712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp362sd2zp
2022-11-23T02:26:17.9710320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp362sd2zp/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9710907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2ayjqzg
2022-11-23T02:26:17.9711571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2ayjqzg/_remote_module_non_scriptable.py
2022-11-23T02:26:17.9712083Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9712560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:26:17.9713100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:26:17.9713575Z INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:26:17.9714045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:26:17.9714584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:26:17.9715317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9716077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:26:17.9716543Z ok (4.086s)
2022-11-23T02:26:17.9716711Z 
2022-11-23T02:26:17.9717017Z ----------------------------------------------------------------------
2022-11-23T02:26:17.9717416Z Ran 1 test in 4.086s
2022-11-23T02:26:17.9717611Z 
2022-11-23T02:26:17.9717738Z OK
2022-11-23T02:26:17.9717899Z 
2022-11-23T02:26:17.9718041Z Generating XML reports...
2022-11-23T02:26:17.9718757Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022613.xml
2022-11-23T02:26:17.9719193Z 
2022-11-23T02:26:17.9719658Z ##[endgroup]
2022-11-23T02:26:17.9720296Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_hjtqpg_e)
2022-11-23T02:26:17.9720674Z 
2022-11-23T02:26:17.9720983Z Running distributed/fsdp/test_fsdp_grad_acc ... [2022-11-23 02:26:17.950740]
2022-11-23T02:26:17.9721742Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:26:17.951046]
2022-11-23T02:27:08.0963395Z 
2022-11-23T02:27:08.0964200Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_grad_acc
2022-11-23T02:27:08.0965260Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_9x4g75s4)
2022-11-23T02:27:08.0966353Z 
2022-11-23T02:27:08.0966497Z Running tests...
2022-11-23T02:27:08.0967368Z ----------------------------------------------------------------------
2022-11-23T02:27:08.0968450Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc
2022-11-23T02:27:08.0969678Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T02:27:08.0970903Z Tests gradient accumulation. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:08.0971792Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63719
2022-11-23T02:27:08.0975497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63720
2022-11-23T02:27:08.0977274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.0978079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.0979094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.0979895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.0980552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.0981091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.0981939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.0984589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.0985453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:08.0986297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:08.0987120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.0987899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.0988492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:08.0989058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:08.0990412Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:27:08.0991355Z   warnings.warn(
2022-11-23T02:27:08.0992604Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:27:08.0993460Z   warnings.warn(
2022-11-23T02:27:08.0993773Z dist init r=0, world=2
2022-11-23T02:27:08.0994093Z dist init r=1, world=2
2022-11-23T02:27:08.0994405Z ok (6.160s)
2022-11-23T02:27:08.0994997Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T02:27:08.0995956Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63802
2022-11-23T02:27:08.0996533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63803
2022-11-23T02:27:08.0997234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.0997750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.0998412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.0998976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.0999621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1000153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1000813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1001368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1001878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:08.1002449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:08.1003264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1004050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1004633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:08.1005188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:08.1007059Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:27:08.1007935Z   warnings.warn(
2022-11-23T02:27:08.1009170Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:27:08.1010029Z   warnings.warn(
2022-11-23T02:27:08.1010359Z dist init r=1, world=2
2022-11-23T02:27:08.1010668Z dist init r=0, world=2
2022-11-23T02:27:08.1010989Z ok (4.412s)
2022-11-23T02:27:08.1011606Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T02:27:08.1012345Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63885
2022-11-23T02:27:08.1012920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63886
2022-11-23T02:27:08.1013610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1014147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1014896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1015446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1016104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1016624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1017266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1017817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1018353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:08.1018914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:08.1019660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1020434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1021039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:08.1021569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:08.1022988Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:27:08.1023872Z   warnings.warn(
2022-11-23T02:27:08.1025099Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:27:08.1025953Z   warnings.warn(
2022-11-23T02:27:08.1026264Z dist init r=1, world=2
2022-11-23T02:27:08.1026587Z dist init r=0, world=2
2022-11-23T02:27:08.1026912Z ok (4.412s)
2022-11-23T02:27:08.1027507Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T02:27:08.1028261Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63968
2022-11-23T02:27:08.1028839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63969
2022-11-23T02:27:08.1029543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1030057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1030711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1031257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1031925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1032432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1033089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1033702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1034217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:08.1034796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:08.1035532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1036310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1036900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:08.1037456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:08.1037884Z dist init r=0, world=2
2022-11-23T02:27:08.1038222Z dist init r=1, world=2
2022-11-23T02:27:08.1038523Z ok (3.210s)
2022-11-23T02:27:08.1039133Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T02:27:08.1039872Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64047
2022-11-23T02:27:08.1040422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64048
2022-11-23T02:27:08.1041183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1041728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1042386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1042920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1043587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1044108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1044756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1045290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1046157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:08.1046750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:08.1047481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1048250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1048859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:08.1049410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:08.1049830Z dist init r=0, world=2
2022-11-23T02:27:08.1050160Z dist init r=1, world=2
2022-11-23T02:27:08.1050480Z ok (3.310s)
2022-11-23T02:27:08.1051082Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T02:27:08.1051839Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64126
2022-11-23T02:27:08.1052410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64127
2022-11-23T02:27:08.1053098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1053718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1054368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1054898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1055554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1056088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1056760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1057302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1057813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:08.1058389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:08.1059123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1059891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1060477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:08.1061106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:08.1061549Z dist init r=1, world=2
2022-11-23T02:27:08.1061861Z dist init r=0, world=2
2022-11-23T02:27:08.1062182Z ok (3.210s)
2022-11-23T02:27:08.1062794Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T02:27:08.1063542Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64205
2022-11-23T02:27:08.1064094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64206
2022-11-23T02:27:08.1064785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1065317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1065977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1066510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1067171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1067705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1068343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1068891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1069423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:08.1069999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:08.1070767Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1071542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1072145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:08.1072694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:08.1074117Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:27:08.1074973Z   warnings.warn(
2022-11-23T02:27:08.1076219Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:27:08.1077078Z   warnings.warn(
2022-11-23T02:27:08.1077404Z dist init r=1, world=2
2022-11-23T02:27:08.1077714Z dist init r=0, world=2
2022-11-23T02:27:08.1078031Z ok (4.512s)
2022-11-23T02:27:08.1078639Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T02:27:08.1079370Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64288
2022-11-23T02:27:08.1079996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64289
2022-11-23T02:27:08.1080703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1081233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1081878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1082426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1083084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1083587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1084240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1084783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1085317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:08.1086196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:08.1086949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1087719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1088320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:08.1088857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:08.1090195Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:27:08.1091155Z   warnings.warn(
2022-11-23T02:27:08.1092381Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:27:08.1093236Z   warnings.warn(
2022-11-23T02:27:08.1093547Z dist init r=1, world=2
2022-11-23T02:27:08.1093873Z dist init r=0, world=2
2022-11-23T02:27:08.1094194Z ok (4.413s)
2022-11-23T02:27:08.1094787Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T02:27:08.1095545Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64371
2022-11-23T02:27:08.1096126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64372
2022-11-23T02:27:08.1096823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1097338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1097990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1098618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1099295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1099801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1100457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1101004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1101518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:08.1102092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:08.1102828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1103599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1104186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:08.1104740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:08.1106086Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:27:08.1106954Z   warnings.warn(
2022-11-23T02:27:08.1108185Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:27:08.1109017Z   warnings.warn(
2022-11-23T02:27:08.1109437Z dist init r=0, world=2
2022-11-23T02:27:08.1109767Z dist init r=1, world=2
2022-11-23T02:27:08.1110068Z ok (4.412s)
2022-11-23T02:27:08.1110675Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T02:27:08.1111417Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64454
2022-11-23T02:27:08.1111993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64455
2022-11-23T02:27:08.1112672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1113197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1113856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1114414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1115060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1115580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1116235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1116760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1117352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:08.1117938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:08.1118675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1119432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1120032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:08.1120586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:08.1121023Z dist init r=0, world=2
2022-11-23T02:27:08.1121339Z dist init r=1, world=2
2022-11-23T02:27:08.1121656Z ok (3.311s)
2022-11-23T02:27:08.1122259Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T02:27:08.1122984Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64533
2022-11-23T02:27:08.1123554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64534
2022-11-23T02:27:08.1124249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1124779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1125419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1126310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1126974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1127489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1128146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1128692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1129321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:08.1129879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:08.1130616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1131386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1131986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:08.1132522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:08.1132958Z dist init r=1, world=2
2022-11-23T02:27:08.1133287Z dist init r=0, world=2
2022-11-23T02:27:08.1133591Z ok (3.211s)
2022-11-23T02:27:08.1134197Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T02:27:08.1134952Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64612
2022-11-23T02:27:08.1135524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64613
2022-11-23T02:27:08.1136197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1136726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1137575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1138134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1138797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:08.1139325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:08.1139979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:08.1140510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:08.1141046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:08.1141618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:08.1142360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1143115Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:08.1143718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:08.1144272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:08.1144685Z dist init r=0, world=2
2022-11-23T02:27:08.1145018Z dist init r=1, world=2
2022-11-23T02:27:08.1145341Z ok (3.210s)
2022-11-23T02:27:08.1145530Z 
2022-11-23T02:27:08.1145847Z ----------------------------------------------------------------------
2022-11-23T02:27:08.1146242Z Ran 12 tests in 47.783s
2022-11-23T02:27:08.1146443Z 
2022-11-23T02:27:08.1146576Z OK
2022-11-23T02:27:08.1146747Z 
2022-11-23T02:27:08.1146910Z Generating XML reports...
2022-11-23T02:27:08.1147551Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20221123022619.xml
2022-11-23T02:27:08.1147929Z 
2022-11-23T02:27:08.1148408Z ##[endgroup]
2022-11-23T02:27:08.1149097Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_9x4g75s4)
2022-11-23T02:27:08.1149569Z 
2022-11-23T02:27:08.1149839Z Running distributed/test_store ... [2022-11-23 02:27:08.096600]
2022-11-23T02:27:08.1150595Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:27:08.096930]
2022-11-23T02:29:13.5561816Z 
2022-11-23T02:29:13.5564852Z Expand the folded group to see the log file of distributed/test_store
2022-11-23T02:29:13.5566416Z ##[group]PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_jvsp76xu)
2022-11-23T02:29:13.5567406Z <unittest.suite.TestSuite tests=[<__main__.FileStoreTest testMethod=test_compare_set>, <__main__.FileStoreTest testMethod=test_init_pg_and_rpc_with_same_file>, <__main__.FileStoreTest testMethod=test_refcount>, <__main__.FileStoreTest testMethod=test_set_get>]>
2022-11-23T02:29:13.5568091Z test_compare_set (__main__.FileStoreTest)
2022-11-23T02:29:13.5568519Z test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest)
2022-11-23T02:29:13.5568933Z test_refcount (__main__.FileStoreTest)
2022-11-23T02:29:13.5569283Z test_set_get (__main__.FileStoreTest)
2022-11-23T02:29:13.5570239Z <unittest.suite.TestSuite tests=[<__main__.HashStoreTest testMethod=test_compare_set>, <__main__.HashStoreTest testMethod=test_set_get>]>
2022-11-23T02:29:13.5571092Z test_compare_set (__main__.HashStoreTest)
2022-11-23T02:29:13.5571454Z test_set_get (__main__.HashStoreTest)
2022-11-23T02:29:13.5573987Z <unittest.suite.TestSuite tests=[<__main__.PrefixFileStoreTest testMethod=test_compare_set>, <__main__.PrefixFileStoreTest testMethod=test_set_get>]>
2022-11-23T02:29:13.5574988Z test_compare_set (__main__.PrefixFileStoreTest)
2022-11-23T02:29:13.5575459Z test_set_get (__main__.PrefixFileStoreTest)
2022-11-23T02:29:13.5575989Z <unittest.suite.TestSuite tests=[<__main__.PrefixStoreTest testMethod=test_get_underlying_store>]>
2022-11-23T02:29:13.5576516Z test_get_underlying_store (__main__.PrefixStoreTest)
2022-11-23T02:29:13.5577159Z <unittest.suite.TestSuite tests=[<__main__.PrefixTCPStoreTest testMethod=test_compare_set>, <__main__.PrefixTCPStoreTest testMethod=test_set_get>]>
2022-11-23T02:29:13.5577769Z test_compare_set (__main__.PrefixTCPStoreTest)
2022-11-23T02:29:13.5578208Z test_set_get (__main__.PrefixTCPStoreTest)
2022-11-23T02:29:13.5578705Z <unittest.suite.TestSuite tests=[<__main__.PythonStoreTest testMethod=test_set_get>]>
2022-11-23T02:29:13.5579200Z test_set_get (__main__.PythonStoreTest)
2022-11-23T02:29:13.5579709Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_nominal>]>
2022-11-23T02:29:13.5580182Z test_nominal (__main__.RendezvousEnvTest)
2022-11-23T02:29:13.5580789Z <unittest.suite.TestSuite tests=[<__main__.RendezvousFileTest testMethod=test_common_errors>, <__main__.RendezvousFileTest testMethod=test_nominal>]>
2022-11-23T02:29:13.5581387Z test_common_errors (__main__.RendezvousFileTest)
2022-11-23T02:29:13.5581824Z test_nominal (__main__.RendezvousFileTest)
2022-11-23T02:29:13.5582592Z <unittest.suite.TestSuite tests=[<__main__.RendezvousTCPTest testMethod=test_common_errors>, <__main__.RendezvousTCPTest testMethod=test_dns_timeout>, <__main__.RendezvousTCPTest testMethod=test_nominal>, <__main__.RendezvousTCPTest testMethod=test_tcp_store_timeout_set>]>
2022-11-23T02:29:13.5583431Z test_common_errors (__main__.RendezvousTCPTest)
2022-11-23T02:29:13.5583868Z test_dns_timeout (__main__.RendezvousTCPTest)
2022-11-23T02:29:13.5584286Z test_nominal (__main__.RendezvousTCPTest)
2022-11-23T02:29:13.5584712Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest)
2022-11-23T02:29:13.5585343Z <unittest.suite.TestSuite tests=[<__main__.RendezvousTest testMethod=test_unknown_handler>, <__main__.RendezvousTest testMethod=test_url_with_node_params>]>
2022-11-23T02:29:13.5585942Z test_unknown_handler (__main__.RendezvousTest)
2022-11-23T02:29:13.5586415Z test_url_with_node_params (__main__.RendezvousTest)
2022-11-23T02:29:13.5587553Z <unittest.suite.TestSuite tests=[<__main__.TCPStoreTest testMethod=test_address_already_in_use>, <__main__.TCPStoreTest testMethod=test_compare_set>, <__main__.TCPStoreTest testMethod=test_init_pg_and_rpc_with_same_socket>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_fixed_world_size>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_nonfixed_world_size>, <__main__.TCPStoreTest testMethod=test_multitenancy>, <__main__.TCPStoreTest testMethod=test_numkeys_delkeys>, <__main__.TCPStoreTest testMethod=test_set_get>]>
2022-11-23T02:29:13.5588759Z test_address_already_in_use (__main__.TCPStoreTest)
2022-11-23T02:29:13.5589189Z test_compare_set (__main__.TCPStoreTest)
2022-11-23T02:29:13.5589628Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest)
2022-11-23T02:29:13.5590107Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest)
2022-11-23T02:29:13.5590597Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest)
2022-11-23T02:29:13.5591037Z test_multitenancy (__main__.TCPStoreTest)
2022-11-23T02:29:13.5591454Z test_numkeys_delkeys (__main__.TCPStoreTest)
2022-11-23T02:29:13.5591863Z test_set_get (__main__.TCPStoreTest)
2022-11-23T02:29:13.5592615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5593175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5593869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5594442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5594727Z 
2022-11-23T02:29:13.5594925Z Running tests...
2022-11-23T02:29:13.5595448Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5596078Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5596655Z   test_compare_set (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5597076Z ok (1.628s)
2022-11-23T02:29:13.5597271Z 
2022-11-23T02:29:13.5597591Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5598011Z Ran 1 test in 1.628s
2022-11-23T02:29:13.5598204Z 
2022-11-23T02:29:13.5598340Z OK
2022-11-23T02:29:13.5598522Z 
2022-11-23T02:29:13.5598691Z Generating XML reports...
2022-11-23T02:29:13.5599354Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022711.xml
2022-11-23T02:29:13.5600132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5600696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5601388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5601969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5602260Z 
2022-11-23T02:29:13.5602393Z Running tests...
2022-11-23T02:29:13.5602899Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5603529Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5604105Z   test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5604715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:13.5605490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:29:13.5606608Z ok (1.659s)
2022-11-23T02:29:13.5606803Z 
2022-11-23T02:29:13.5607104Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5607517Z Ran 1 test in 1.659s
2022-11-23T02:29:13.5607722Z 
2022-11-23T02:29:13.5607852Z OK
2022-11-23T02:29:13.5608026Z 
2022-11-23T02:29:13.5608172Z Generating XML reports...
2022-11-23T02:29:13.5608932Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022715.xml
2022-11-23T02:29:13.5609684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5610220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5610855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5611409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5611682Z 
2022-11-23T02:29:13.5611834Z Running tests...
2022-11-23T02:29:13.5612328Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5612910Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5613448Z   test_refcount (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5613868Z ok (1.659s)
2022-11-23T02:29:13.5614037Z 
2022-11-23T02:29:13.5614344Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5614750Z Ran 1 test in 1.660s
2022-11-23T02:29:13.5614950Z 
2022-11-23T02:29:13.5615082Z OK
2022-11-23T02:29:13.5615257Z 
2022-11-23T02:29:13.5615402Z Generating XML reports...
2022-11-23T02:29:13.5616036Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022719.xml
2022-11-23T02:29:13.5616791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5617403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5618056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5618607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5619062Z 
2022-11-23T02:29:13.5619214Z Running tests...
2022-11-23T02:29:13.5619696Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5620278Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5620814Z   test_set_get (__main__.FileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5621225Z ok (1.645s)
2022-11-23T02:29:13.5621414Z 
2022-11-23T02:29:13.5621702Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5622110Z Ran 1 test in 1.646s
2022-11-23T02:29:13.5622310Z 
2022-11-23T02:29:13.5622442Z OK
2022-11-23T02:29:13.5622617Z 
2022-11-23T02:29:13.5622763Z Generating XML reports...
2022-11-23T02:29:13.5623397Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022723.xml
2022-11-23T02:29:13.5624145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5624681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5625315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5625866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5626137Z 
2022-11-23T02:29:13.5626287Z Running tests...
2022-11-23T02:29:13.5626751Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5627346Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5627897Z   test_compare_set (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5628320Z ok (1.633s)
2022-11-23T02:29:13.5628487Z 
2022-11-23T02:29:13.5628793Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5629196Z Ran 1 test in 1.634s
2022-11-23T02:29:13.5629397Z 
2022-11-23T02:29:13.5629593Z OK
2022-11-23T02:29:13.5629771Z 
2022-11-23T02:29:13.5629915Z Generating XML reports...
2022-11-23T02:29:13.5630547Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022727.xml
2022-11-23T02:29:13.5631297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5631828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5632465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5633020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5633296Z 
2022-11-23T02:29:13.5633445Z Running tests...
2022-11-23T02:29:13.5633912Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5634509Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5635050Z   test_set_get (__main__.HashStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5635460Z ok (1.644s)
2022-11-23T02:29:13.5635626Z 
2022-11-23T02:29:13.5635930Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5636330Z Ran 1 test in 1.644s
2022-11-23T02:29:13.5636529Z 
2022-11-23T02:29:13.5636660Z OK
2022-11-23T02:29:13.5636836Z 
2022-11-23T02:29:13.5636981Z Generating XML reports...
2022-11-23T02:29:13.5637611Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022731.xml
2022-11-23T02:29:13.5638424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5638962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5639604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5640156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5640427Z 
2022-11-23T02:29:13.5640576Z Running tests...
2022-11-23T02:29:13.5641043Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5641640Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5642196Z   test_compare_set (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5642629Z ok (1.612s)
2022-11-23T02:29:13.5642798Z 
2022-11-23T02:29:13.5643100Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5643502Z Ran 1 test in 1.612s
2022-11-23T02:29:13.5643703Z 
2022-11-23T02:29:13.5643834Z OK
2022-11-23T02:29:13.5644009Z 
2022-11-23T02:29:13.5644153Z Generating XML reports...
2022-11-23T02:29:13.5644806Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022735.xml
2022-11-23T02:29:13.5645583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5646787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5647435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5647983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5648253Z 
2022-11-23T02:29:13.5648402Z Running tests...
2022-11-23T02:29:13.5648871Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5649477Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5650029Z   test_set_get (__main__.PrefixFileStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5650456Z ok (1.636s)
2022-11-23T02:29:13.5650623Z 
2022-11-23T02:29:13.5650928Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5651446Z Ran 1 test in 1.637s
2022-11-23T02:29:13.5651647Z 
2022-11-23T02:29:13.5651779Z OK
2022-11-23T02:29:13.5651955Z 
2022-11-23T02:29:13.5652101Z Generating XML reports...
2022-11-23T02:29:13.5652754Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022739.xml
2022-11-23T02:29:13.5653525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5654055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5654697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5655248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5655521Z 
2022-11-23T02:29:13.5655672Z Running tests...
2022-11-23T02:29:13.5656134Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5656737Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5657250Z   test_get_underlying_store (__main__.PrefixStoreTest) ... ok (0.003s)
2022-11-23T02:29:13.5657518Z 
2022-11-23T02:29:13.5657823Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5658204Z Ran 1 test in 0.003s
2022-11-23T02:29:13.5658406Z 
2022-11-23T02:29:13.5658538Z OK
2022-11-23T02:29:13.5658711Z 
2022-11-23T02:29:13.5658875Z Generating XML reports...
2022-11-23T02:29:13.5659571Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221123022742.xml
2022-11-23T02:29:13.5660344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5660879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5661575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5662115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5662384Z 
2022-11-23T02:29:13.5662533Z Running tests...
2022-11-23T02:29:13.5663012Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5663595Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5664154Z   test_compare_set (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5664583Z ok (1.667s)
2022-11-23T02:29:13.5664771Z 
2022-11-23T02:29:13.5665081Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5665466Z Ran 1 test in 1.668s
2022-11-23T02:29:13.5665667Z 
2022-11-23T02:29:13.5665799Z OK
2022-11-23T02:29:13.5665967Z 
2022-11-23T02:29:13.5666130Z Generating XML reports...
2022-11-23T02:29:13.5666768Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022744.xml
2022-11-23T02:29:13.5667548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5668075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5668727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5669260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5669528Z 
2022-11-23T02:29:13.5669677Z Running tests...
2022-11-23T02:29:13.5670158Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5670746Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5671288Z   test_set_get (__main__.PrefixTCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5671714Z ok (1.612s)
2022-11-23T02:29:13.5671905Z 
2022-11-23T02:29:13.5672272Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5672658Z Ran 1 test in 1.613s
2022-11-23T02:29:13.5672859Z 
2022-11-23T02:29:13.5672992Z OK
2022-11-23T02:29:13.5673167Z 
2022-11-23T02:29:13.5673331Z Generating XML reports...
2022-11-23T02:29:13.5673963Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022748.xml
2022-11-23T02:29:13.5674727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5675259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5675896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5676447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5676718Z 
2022-11-23T02:29:13.5676869Z Running tests...
2022-11-23T02:29:13.5677354Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5677931Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5678474Z   test_set_get (__main__.PythonStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5678889Z ok (1.619s)
2022-11-23T02:29:13.5679077Z 
2022-11-23T02:29:13.5679363Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5679766Z Ran 1 test in 1.619s
2022-11-23T02:29:13.5679966Z 
2022-11-23T02:29:13.5680097Z OK
2022-11-23T02:29:13.5680269Z 
2022-11-23T02:29:13.5680434Z Generating XML reports...
2022-11-23T02:29:13.5681111Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221123022752.xml
2022-11-23T02:29:13.5681872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5682402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5683040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5683588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5683858Z 
2022-11-23T02:29:13.5684006Z Running tests...
2022-11-23T02:29:13.5684488Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5685068Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5685612Z   test_nominal (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5686737Z ok (1.634s)
2022-11-23T02:29:13.5686928Z 
2022-11-23T02:29:13.5687217Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5687621Z Ran 1 test in 1.634s
2022-11-23T02:29:13.5687824Z 
2022-11-23T02:29:13.5687957Z OK
2022-11-23T02:29:13.5688130Z 
2022-11-23T02:29:13.5688294Z Generating XML reports...
2022-11-23T02:29:13.5688926Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221123022756.xml
2022-11-23T02:29:13.5689683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5690216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5690852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5691404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5691675Z 
2022-11-23T02:29:13.5691829Z Running tests...
2022-11-23T02:29:13.5692312Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5692889Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5693439Z   test_common_errors (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5693989Z ok (1.648s)
2022-11-23T02:29:13.5694175Z 
2022-11-23T02:29:13.5694459Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5694865Z Ran 1 test in 1.648s
2022-11-23T02:29:13.5695068Z 
2022-11-23T02:29:13.5695199Z OK
2022-11-23T02:29:13.5695372Z 
2022-11-23T02:29:13.5695535Z Generating XML reports...
2022-11-23T02:29:13.5696165Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022800.xml
2022-11-23T02:29:13.5696936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5697466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5698101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5698652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5698928Z 
2022-11-23T02:29:13.5699078Z Running tests...
2022-11-23T02:29:13.5699567Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5700144Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5700686Z   test_nominal (__main__.RendezvousFileTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5701109Z ok (1.651s)
2022-11-23T02:29:13.5701297Z 
2022-11-23T02:29:13.5701581Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5701984Z Ran 1 test in 1.651s
2022-11-23T02:29:13.5702184Z 
2022-11-23T02:29:13.5702386Z OK
2022-11-23T02:29:13.5702565Z 
2022-11-23T02:29:13.5702731Z Generating XML reports...
2022-11-23T02:29:13.5703362Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022804.xml
2022-11-23T02:29:13.5704124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5704663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5705296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5705844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5706111Z 
2022-11-23T02:29:13.5706261Z Running tests...
2022-11-23T02:29:13.5706743Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5707323Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5707876Z   test_common_errors (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5708305Z ok (1.592s)
2022-11-23T02:29:13.5708492Z 
2022-11-23T02:29:13.5708777Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5709186Z Ran 1 test in 1.592s
2022-11-23T02:29:13.5709392Z 
2022-11-23T02:29:13.5709524Z OK
2022-11-23T02:29:13.5709695Z 
2022-11-23T02:29:13.5709839Z Generating XML reports...
2022-11-23T02:29:13.5710482Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022808.xml
2022-11-23T02:29:13.5711247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5711776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5712412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5712965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5713238Z 
2022-11-23T02:29:13.5713387Z Running tests...
2022-11-23T02:29:13.5713869Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5714451Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5715066Z   test_dns_timeout (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5715802Z [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known).
2022-11-23T02:29:13.5716396Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456).
2022-11-23T02:29:13.5716835Z ok (1.647s)
2022-11-23T02:29:13.5717023Z 
2022-11-23T02:29:13.5717328Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5717736Z Ran 1 test in 1.647s
2022-11-23T02:29:13.5717920Z 
2022-11-23T02:29:13.5718052Z OK
2022-11-23T02:29:13.5718224Z 
2022-11-23T02:29:13.5718387Z Generating XML reports...
2022-11-23T02:29:13.5719030Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022812.xml
2022-11-23T02:29:13.5719777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5720306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5720960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5721511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5721762Z 
2022-11-23T02:29:13.5721914Z Running tests...
2022-11-23T02:29:13.5722397Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5723048Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5723584Z   test_nominal (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5724008Z ok (1.597s)
2022-11-23T02:29:13.5724198Z 
2022-11-23T02:29:13.5724497Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5724905Z Ran 1 test in 1.597s
2022-11-23T02:29:13.5725086Z 
2022-11-23T02:29:13.5725220Z OK
2022-11-23T02:29:13.5725395Z 
2022-11-23T02:29:13.5725557Z Generating XML reports...
2022-11-23T02:29:13.5726593Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022816.xml
2022-11-23T02:29:13.5727336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5727866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5728527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5729080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5729327Z 
2022-11-23T02:29:13.5729478Z Running tests...
2022-11-23T02:29:13.5729954Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5730557Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5731105Z   test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5731541Z ok (11.866s)
2022-11-23T02:29:13.5731730Z 
2022-11-23T02:29:13.5732034Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5732444Z Ran 1 test in 11.867s
2022-11-23T02:29:13.5732628Z 
2022-11-23T02:29:13.5732764Z OK
2022-11-23T02:29:13.5732935Z 
2022-11-23T02:29:13.5733100Z Generating XML reports...
2022-11-23T02:29:13.5733753Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022819.xml
2022-11-23T02:29:13.5734498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5735034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5735689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5736334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5736584Z 
2022-11-23T02:29:13.5736735Z Running tests...
2022-11-23T02:29:13.5737216Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5737814Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5738346Z   test_unknown_handler (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5738773Z ok (1.640s)
2022-11-23T02:29:13.5738963Z 
2022-11-23T02:29:13.5739270Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5739654Z Ran 1 test in 1.641s
2022-11-23T02:29:13.5739857Z 
2022-11-23T02:29:13.5739987Z OK
2022-11-23T02:29:13.5740161Z 
2022-11-23T02:29:13.5740324Z Generating XML reports...
2022-11-23T02:29:13.5740957Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123022834.xml
2022-11-23T02:29:13.5741702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5742235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5742889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5743441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5743693Z 
2022-11-23T02:29:13.5743846Z Running tests...
2022-11-23T02:29:13.5744394Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5744999Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5745536Z   test_url_with_node_params (__main__.RendezvousTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5745970Z ok (1.622s)
2022-11-23T02:29:13.5746158Z 
2022-11-23T02:29:13.5746459Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5746845Z Ran 1 test in 1.622s
2022-11-23T02:29:13.5747044Z 
2022-11-23T02:29:13.5747180Z OK
2022-11-23T02:29:13.5747353Z 
2022-11-23T02:29:13.5747517Z Generating XML reports...
2022-11-23T02:29:13.5748145Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123022837.xml
2022-11-23T02:29:13.5748883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5749415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5750067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5750599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5750869Z 
2022-11-23T02:29:13.5751024Z Running tests...
2022-11-23T02:29:13.5751506Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5752104Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5752645Z   test_address_already_in_use (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5753340Z [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:39459 (errno: 98 - Address already in use).
2022-11-23T02:29:13.5754017Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:39459 (errno: 98 - Address already in use).
2022-11-23T02:29:13.5754563Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2022-11-23T02:29:13.5754961Z ok (1.655s)
2022-11-23T02:29:13.5755151Z 
2022-11-23T02:29:13.5755457Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5755863Z Ran 1 test in 1.655s
2022-11-23T02:29:13.5756138Z 
2022-11-23T02:29:13.5756251Z OK
2022-11-23T02:29:13.5756423Z 
2022-11-23T02:29:13.5756585Z Generating XML reports...
2022-11-23T02:29:13.5757217Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022841.xml
2022-11-23T02:29:13.5757966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5758479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5759133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5759685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5759956Z 
2022-11-23T02:29:13.5760086Z Running tests...
2022-11-23T02:29:13.5760561Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5761163Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5761751Z   test_compare_set (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5762154Z ok (1.651s)
2022-11-23T02:29:13.5762343Z 
2022-11-23T02:29:13.5762644Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5763050Z Ran 1 test in 1.651s
2022-11-23T02:29:13.5763230Z 
2022-11-23T02:29:13.5763362Z OK
2022-11-23T02:29:13.5763534Z 
2022-11-23T02:29:13.5763695Z Generating XML reports...
2022-11-23T02:29:13.5764321Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022845.xml
2022-11-23T02:29:13.5765104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5765642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5766619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5767176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5767446Z 
2022-11-23T02:29:13.5767575Z Running tests...
2022-11-23T02:29:13.5768057Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5768654Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5769209Z   test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5769795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:13.5770535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:29:13.5771020Z ok (1.729s)
2022-11-23T02:29:13.5771208Z 
2022-11-23T02:29:13.5771495Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5771897Z Ran 1 test in 1.729s
2022-11-23T02:29:13.5772104Z 
2022-11-23T02:29:13.5772237Z OK
2022-11-23T02:29:13.5772409Z 
2022-11-23T02:29:13.5772573Z Generating XML reports...
2022-11-23T02:29:13.5773187Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022849.xml
2022-11-23T02:29:13.5773939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5774471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5775111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5775668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5775940Z 
2022-11-23T02:29:13.5776090Z Running tests...
2022-11-23T02:29:13.5776578Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5777154Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5777818Z   test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5778261Z ok (1.672s)
2022-11-23T02:29:13.5778450Z 
2022-11-23T02:29:13.5778735Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5779139Z Ran 1 test in 1.673s
2022-11-23T02:29:13.5779337Z 
2022-11-23T02:29:13.5779468Z OK
2022-11-23T02:29:13.5779642Z 
2022-11-23T02:29:13.5779787Z Generating XML reports...
2022-11-23T02:29:13.5780412Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022853.xml
2022-11-23T02:29:13.5781163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5781693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5782327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5782881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5783152Z 
2022-11-23T02:29:13.5783302Z Running tests...
2022-11-23T02:29:13.5783785Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5784360Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5784932Z   test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5785379Z ok (1.614s)
2022-11-23T02:29:13.5785566Z 
2022-11-23T02:29:13.5785939Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5786352Z Ran 1 test in 1.615s
2022-11-23T02:29:13.5786553Z 
2022-11-23T02:29:13.5786684Z OK
2022-11-23T02:29:13.5786855Z 
2022-11-23T02:29:13.5786999Z Generating XML reports...
2022-11-23T02:29:13.5787623Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022857.xml
2022-11-23T02:29:13.5788375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5788905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5789543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5790087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5790358Z 
2022-11-23T02:29:13.5790507Z Running tests...
2022-11-23T02:29:13.5790975Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5791574Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5792117Z   test_multitenancy (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5792535Z ok (1.653s)
2022-11-23T02:29:13.5792700Z 
2022-11-23T02:29:13.5793008Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5793414Z Ran 1 test in 1.654s
2022-11-23T02:29:13.5793616Z 
2022-11-23T02:29:13.5793748Z OK
2022-11-23T02:29:13.5793923Z 
2022-11-23T02:29:13.5794067Z Generating XML reports...
2022-11-23T02:29:13.5794689Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022901.xml
2022-11-23T02:29:13.5795437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5795967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5796603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5797152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5797421Z 
2022-11-23T02:29:13.5797570Z Running tests...
2022-11-23T02:29:13.5798096Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5798692Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5799232Z   test_numkeys_delkeys (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5799653Z ok (3.633s)
2022-11-23T02:29:13.5799822Z 
2022-11-23T02:29:13.5800128Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5800540Z Ran 1 test in 3.634s
2022-11-23T02:29:13.5800741Z 
2022-11-23T02:29:13.5800873Z OK
2022-11-23T02:29:13.5801046Z 
2022-11-23T02:29:13.5801190Z Generating XML reports...
2022-11-23T02:29:13.5801824Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022905.xml
2022-11-23T02:29:13.5802575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:13.5803106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:13.5803753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:13.5804307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:13.5804577Z 
2022-11-23T02:29:13.5804727Z Running tests...
2022-11-23T02:29:13.5805189Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5806101Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2022-11-23T02:29:13.5806641Z   test_set_get (__main__.TCPStoreTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:13.5807130Z ok (1.646s)
2022-11-23T02:29:13.5807306Z 
2022-11-23T02:29:13.5807622Z ----------------------------------------------------------------------
2022-11-23T02:29:13.5808030Z Ran 1 test in 1.646s
2022-11-23T02:29:13.5808229Z 
2022-11-23T02:29:13.5808361Z OK
2022-11-23T02:29:13.5808534Z 
2022-11-23T02:29:13.5808678Z Generating XML reports...
2022-11-23T02:29:13.5809305Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022911.xml
2022-11-23T02:29:13.5809665Z 
2022-11-23T02:29:13.5810136Z ##[endgroup]
2022-11-23T02:29:13.5810747Z FINISHED PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_jvsp76xu)
2022-11-23T02:29:13.5811112Z 
2022-11-23T02:29:13.5811406Z Running distributed/fsdp/test_wrap ... [2022-11-23 02:29:13.556851]
2022-11-23T02:29:13.5812147Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:29:13.557135]
2022-11-23T02:30:40.8632050Z 
2022-11-23T02:30:40.8633115Z Expand the folded group to see the log file of distributed/fsdp/test_wrap
2022-11-23T02:30:40.8634493Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_mxucm99q)
2022-11-23T02:30:40.8634925Z 
2022-11-23T02:30:40.8635064Z Running tests...
2022-11-23T02:30:40.8635632Z ----------------------------------------------------------------------
2022-11-23T02:30:40.8636262Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap
2022-11-23T02:30:40.8636746Z   test_always_wrap (__main__.TestAutoWrap)
2022-11-23T02:30:40.8637198Z Test to ensure that if `always_wrap_policy` is ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:40.8637617Z ok (1.638s)
2022-11-23T02:30:40.8639126Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8640674Z   warnings.warn(
2022-11-23T02:30:40.8641064Z ok (0.005s)
2022-11-23T02:30:40.8641487Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.005s)
2022-11-23T02:30:40.8642184Z   test_auto_wrap_api (__main__.TestAutoWrap)
2022-11-23T02:30:40.8642672Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... ok (0.003s)
2022-11-23T02:30:40.8643398Z   test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap)
2022-11-23T02:30:40.8643939Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... ok (0.002s)
2022-11-23T02:30:40.8644487Z   test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap)
2022-11-23T02:30:40.8645029Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... ok (0.002s)
2022-11-23T02:30:40.8648882Z   test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap)
2022-11-23T02:30:40.8649593Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... ok (0.003s)
2022-11-23T02:30:40.8650129Z   test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap)
2022-11-23T02:30:40.8650681Z Test to ensure force-leaf modules are not wrapped. ... ok (0.002s)
2022-11-23T02:30:40.8651409Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8652609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:30:40.8653151Z ok (0.466s)
2022-11-23T02:30:40.8653816Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8654767Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:30:40.8655300Z ok (0.043s)
2022-11-23T02:30:40.8655815Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... ok (0.002s)
2022-11-23T02:30:40.8656548Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... ok (0.002s)
2022-11-23T02:30:40.8657409Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8658353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:30:40.8659501Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:30:40.8660819Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:30:40.8661518Z ok (0.061s)
2022-11-23T02:30:40.8662175Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8663239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:30:40.8663719Z ok (0.042s)
2022-11-23T02:30:40.8664356Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8665288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:30:40.8665761Z ok (0.043s)
2022-11-23T02:30:40.8666407Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8667314Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:30:40.8668434Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:30:40.8669816Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:30:40.8671144Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:30:40.8672479Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:30:40.8673152Z ok (0.072s)
2022-11-23T02:30:40.8673594Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.003s)
2022-11-23T02:30:40.8674176Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.003s)
2022-11-23T02:30:40.8674670Z   test_module_wrap_policy (__main__.TestAutoWrap)
2022-11-23T02:30:40.8675070Z Tests the ``ModuleWrapPolicy``. ... ok (0.026s)
2022-11-23T02:30:40.8675506Z   test_transformer_auto_wrap_policy (__main__.TestAutoWrap)
2022-11-23T02:30:40.8675948Z Tests the ``transformer_auto_wrap_policy``. ... ok (0.020s)
2022-11-23T02:30:40.8676407Z   test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... ok (0.002s)
2022-11-23T02:30:40.8676868Z   test_wrap_override_defaults (__main__.TestAutoWrap) ... ok (0.002s)
2022-11-23T02:30:40.8677358Z   test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.002s)
2022-11-23T02:30:40.8677867Z   test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.002s)
2022-11-23T02:30:40.8678546Z   test_bn_always_wrapped_individually (__main__.TestFSDPWrap)
2022-11-23T02:30:40.8679160Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65903
2022-11-23T02:30:40.8679784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65904
2022-11-23T02:30:40.8680563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8681077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8681737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8682285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8682947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8683457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8684106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8684648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8685162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8686073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8686826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8687592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8688175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8688813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8690175Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8691031Z   warnings.warn(
2022-11-23T02:30:40.8692268Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8693107Z   warnings.warn(
2022-11-23T02:30:40.8693432Z dist init r=1, world=2
2022-11-23T02:30:40.8693762Z dist init r=0, world=2
2022-11-23T02:30:40.8694059Z ok (3.311s)
2022-11-23T02:30:40.8694512Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-11-23T02:30:40.8695165Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65982
2022-11-23T02:30:40.8695779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65983
2022-11-23T02:30:40.8696459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8696989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8697645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8698193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8698831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8699353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8700096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8700619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8701149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8701721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8702460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8703213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8703813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8704356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8705704Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8706548Z   warnings.warn(
2022-11-23T02:30:40.8707836Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8708706Z   warnings.warn(
2022-11-23T02:30:40.8709037Z dist init r=1, world=2
2022-11-23T02:30:40.8709345Z dist init r=0, world=2
2022-11-23T02:30:40.8709663Z ok (3.309s)
2022-11-23T02:30:40.8710115Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-11-23T02:30:40.8710761Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66061
2022-11-23T02:30:40.8711361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66062
2022-11-23T02:30:40.8712060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8712585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8713240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8713779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8714437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8714956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8715584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8716127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8716662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8717232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8717953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8718800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8719398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8719946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8720363Z dist init r=0, world=2
2022-11-23T02:30:40.8720695Z dist init r=1, world=2
2022-11-23T02:30:40.8721016Z ok (3.309s)
2022-11-23T02:30:40.8721454Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2022-11-23T02:30:40.8722103Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66140
2022-11-23T02:30:40.8722716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66141
2022-11-23T02:30:40.8723397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8723923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8724574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8725116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8726131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8726757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8727423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8727967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8728478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8729056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8729788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8730538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8731134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8731686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8733035Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8733901Z   warnings.warn(
2022-11-23T02:30:40.8735124Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8735970Z   warnings.warn(
2022-11-23T02:30:40.8736297Z dist init r=1, world=2
2022-11-23T02:30:40.8736623Z dist init r=0, world=2
2022-11-23T02:30:40.8736924Z ok (3.309s)
2022-11-23T02:30:40.8737372Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2022-11-23T02:30:40.8738106Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66219
2022-11-23T02:30:40.8738700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66220
2022-11-23T02:30:40.8739392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8739923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8740580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8741116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8741773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8742292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8742933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8743477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8744007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8744578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8745346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8746128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8746726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8747273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8747695Z dist init r=0, world=2
2022-11-23T02:30:40.8748025Z dist init r=1, world=2
2022-11-23T02:30:40.8748345Z ok (3.309s)
2022-11-23T02:30:40.8749041Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66298
2022-11-23T02:30:40.8749833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66299
2022-11-23T02:30:40.8750527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8751054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8751689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8752243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8752901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8753423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8754052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8754595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8755127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8755730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8756473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8757305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8757904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8758431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8759770Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8760633Z   warnings.warn(
2022-11-23T02:30:40.8761866Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8762715Z   warnings.warn(
2022-11-23T02:30:40.8763027Z dist init r=1, world=2
2022-11-23T02:30:40.8763350Z dist init r=0, world=2
2022-11-23T02:30:40.8763666Z ok (3.812s)
2022-11-23T02:30:40.8764432Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66381
2022-11-23T02:30:40.8765243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66382
2022-11-23T02:30:40.8766213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8766746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8767385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8767931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8768585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8769111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8769747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8770287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8770814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8771379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8772117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8772878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8773477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8774014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8774449Z dist init r=1, world=2
2022-11-23T02:30:40.8774778Z dist init r=0, world=2
2022-11-23T02:30:40.8775079Z ok (3.912s)
2022-11-23T02:30:40.8775788Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66464
2022-11-23T02:30:40.8776676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66465
2022-11-23T02:30:40.8777367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8777874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8778513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8779043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8779691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8780221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8780893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8781432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8781957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8782512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8783243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8784070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8784664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8785211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8786557Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8787404Z   warnings.warn(
2022-11-23T02:30:40.8788643Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8789497Z   warnings.warn(
2022-11-23T02:30:40.8789807Z dist init r=1, world=2
2022-11-23T02:30:40.8790131Z dist init r=0, world=2
2022-11-23T02:30:40.8790446Z ok (3.812s)
2022-11-23T02:30:40.8791132Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66547
2022-11-23T02:30:40.8791920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66548
2022-11-23T02:30:40.8792610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8793139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8793772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8794383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8795043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8795545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8796187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8796727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8797256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8797813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8798551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8799315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8799917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8800447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8800873Z dist init r=1, world=2
2022-11-23T02:30:40.8801200Z dist init r=0, world=2
2022-11-23T02:30:40.8801497Z ok (3.812s)
2022-11-23T02:30:40.8802249Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66630
2022-11-23T02:30:40.8803044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66631
2022-11-23T02:30:40.8803732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8804246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8804898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8805438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8806409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8806932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8807585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8808120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8808630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8809202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8809934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8810694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8811272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8811823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8813171Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8814122Z   warnings.warn(
2022-11-23T02:30:40.8815344Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8816187Z   warnings.warn(
2022-11-23T02:30:40.8816522Z dist init r=0, world=2
2022-11-23T02:30:40.8816846Z dist init r=1, world=2
2022-11-23T02:30:40.8817147Z ok (3.812s)
2022-11-23T02:30:40.8817859Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66713
2022-11-23T02:30:40.8818657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66714
2022-11-23T02:30:40.8819351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8819863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8820518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8821127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8821797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8822299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8822945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8823491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8823999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8824572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8825307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8826069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8826652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8827197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8827632Z dist init r=1, world=2
2022-11-23T02:30:40.8827945Z dist init r=0, world=2
2022-11-23T02:30:40.8828262Z ok (3.912s)
2022-11-23T02:30:40.8828966Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66796
2022-11-23T02:30:40.8829756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66797
2022-11-23T02:30:40.8830433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8830960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8831610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8832158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8832881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8833404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8834047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8834570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8835101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8835682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8836420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8837173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8837777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8838327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8839719Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8840587Z   warnings.warn(
2022-11-23T02:30:40.8841804Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8842664Z   warnings.warn(
2022-11-23T02:30:40.8842992Z dist init r=1, world=2
2022-11-23T02:30:40.8843319Z dist init r=0, world=2
2022-11-23T02:30:40.8843619Z ok (3.812s)
2022-11-23T02:30:40.8844327Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66879
2022-11-23T02:30:40.8845116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66880
2022-11-23T02:30:40.8846032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8846580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8847240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8847785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8848424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8848951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8849608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8850150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8850660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8851224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8852059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8852809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8853409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8853958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8854390Z dist init r=1, world=2
2022-11-23T02:30:40.8854705Z dist init r=0, world=2
2022-11-23T02:30:40.8855018Z ok (3.812s)
2022-11-23T02:30:40.8855772Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66962
2022-11-23T02:30:40.8856567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66963
2022-11-23T02:30:40.8857245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8857769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8858418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8858947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8859669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8860204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8860856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8861384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8861911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8862480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8863195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8863965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8864567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8865113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8865529Z dist init r=1, world=2
2022-11-23T02:30:40.8865857Z dist init r=0, world=2
2022-11-23T02:30:40.8866177Z ok (3.311s)
2022-11-23T02:30:40.8866866Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67041
2022-11-23T02:30:40.8867658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67042
2022-11-23T02:30:40.8868350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8868882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8869519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8870067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8870720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8871304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8871942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8872481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8873010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8873565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8874305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8875074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8875681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8876213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8876642Z dist init r=1, world=2
2022-11-23T02:30:40.8876970Z dist init r=0, world=2
2022-11-23T02:30:40.8877270Z ok (3.912s)
2022-11-23T02:30:40.8877979Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67124
2022-11-23T02:30:40.8878821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67125
2022-11-23T02:30:40.8879521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8880030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8880690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8881233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8881885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8882387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8883036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8883582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8884113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8884667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8885401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8886551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8887137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8887681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8888108Z dist init r=1, world=2
2022-11-23T02:30:40.8888436Z dist init r=0, world=2
2022-11-23T02:30:40.8888734Z ok (3.311s)
2022-11-23T02:30:40.8889443Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67203
2022-11-23T02:30:40.8890229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67204
2022-11-23T02:30:40.8891004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8891533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8892183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8892724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8893365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8893892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8894541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8895076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8895593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8896164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8896900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8897647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8898333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8898895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8899331Z dist init r=0, world=2
2022-11-23T02:30:40.8899641Z dist init r=1, world=2
2022-11-23T02:30:40.8899959Z ok (3.812s)
2022-11-23T02:30:40.8900661Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67286
2022-11-23T02:30:40.8901451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67287
2022-11-23T02:30:40.8902130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8902657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8903309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8903839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8904495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8905020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8905671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8906192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8906722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8907298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8908019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8908789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8909382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8909990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8910406Z dist init r=0, world=2
2022-11-23T02:30:40.8910730Z dist init r=1, world=2
2022-11-23T02:30:40.8911047Z ok (3.311s)
2022-11-23T02:30:40.8911728Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67365
2022-11-23T02:30:40.8912518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67366
2022-11-23T02:30:40.8913212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8913738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8914372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8914922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8915580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8916100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8916735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8917275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8917854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8918437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8919162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8919920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8920528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8921074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8921503Z dist init r=0, world=2
2022-11-23T02:30:40.8921814Z dist init r=1, world=2
2022-11-23T02:30:40.8922127Z ok (3.812s)
2022-11-23T02:30:40.8922834Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67448
2022-11-23T02:30:40.8923615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67449
2022-11-23T02:30:40.8924288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8924814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8925463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8926277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8926935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8927454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8928107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8928632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8929157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8929823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8930542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8931305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8931901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8932447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8932865Z dist init r=0, world=2
2022-11-23T02:30:40.8933195Z dist init r=1, world=2
2022-11-23T02:30:40.8933511Z ok (3.311s)
2022-11-23T02:30:40.8934199Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67527
2022-11-23T02:30:40.8934990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67528
2022-11-23T02:30:40.8935676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8936197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8936833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8937448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8938118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8938635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8939264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8939807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8940332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8940887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8941618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8942386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8942982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8943512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8943933Z dist init r=1, world=2
2022-11-23T02:30:40.8944263Z dist init r=0, world=2
2022-11-23T02:30:40.8944568Z ok (3.812s)
2022-11-23T02:30:40.8945100Z   test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67610
2022-11-23T02:30:40.8945725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67611
2022-11-23T02:30:40.8946409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8946918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8947570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8948177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8948834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8949426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8950076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8950610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8951121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8951686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8952421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8953186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8953764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8954314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8955701Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8956548Z   warnings.warn(
2022-11-23T02:30:40.8957814Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8958668Z   warnings.warn(
2022-11-23T02:30:40.8958986Z dist init r=0, world=2
2022-11-23T02:30:40.8959303Z dist init r=1, world=2
2022-11-23T02:30:40.8959600Z ok (3.309s)
2022-11-23T02:30:40.8960128Z   test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67689
2022-11-23T02:30:40.8960750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67690
2022-11-23T02:30:40.8961442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8961949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8962593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8963142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8963797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:40.8964302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:40.8964945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:40.8965478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:40.8966173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:40.8966745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:40.8967480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8968346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:40.8968929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:40.8969475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:40.8970828Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8971686Z   warnings.warn(
2022-11-23T02:30:40.8972909Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:40.8973751Z   warnings.warn(
2022-11-23T02:30:40.8974072Z dist init r=0, world=2
2022-11-23T02:30:40.8974395Z dist init r=1, world=2
2022-11-23T02:30:40.8974693Z ok (3.309s)
2022-11-23T02:30:40.8974877Z 
2022-11-23T02:30:40.8975256Z ----------------------------------------------------------------------
2022-11-23T02:30:40.8975670Z Ran 47 tests in 84.913s
2022-11-23T02:30:40.8975870Z 
2022-11-23T02:30:40.8976000Z OK
2022-11-23T02:30:40.8976153Z 
2022-11-23T02:30:40.8976314Z Generating XML reports...
2022-11-23T02:30:40.8976956Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221123022915.xml
2022-11-23T02:30:40.8977744Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221123022915.xml
2022-11-23T02:30:40.8978109Z 
2022-11-23T02:30:40.8978570Z ##[endgroup]
2022-11-23T02:30:40.8979220Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_mxucm99q)
2022-11-23T02:30:40.8979595Z 
2022-11-23T02:30:40.8979946Z Running distributed/_shard/sharded_tensor/test_sharded_tensor ... [2022-11-23 02:30:40.863696]
2022-11-23T02:30:40.8980768Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:30:40.863976]
2022-11-23T02:33:12.8355692Z 
2022-11-23T02:33:12.8356183Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor
2022-11-23T02:33:12.8357322Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_f5ukg2fs)
2022-11-23T02:33:12.8362101Z 
2022-11-23T02:33:12.8362916Z Running tests...
2022-11-23T02:33:12.8363609Z ----------------------------------------------------------------------
2022-11-23T02:33:12.8364576Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor
2022-11-23T02:33:12.8365226Z   test_empty (__main__.TestCreateTensorFromParams) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:33:12.8366136Z ok (1.614s)
2022-11-23T02:33:12.8366684Z   test_local_tensor (__main__.TestLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67803
2022-11-23T02:33:12.8367522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67804
2022-11-23T02:33:12.8368064Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67805
2022-11-23T02:33:12.8369029Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67806
2022-11-23T02:33:12.8370002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8370565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8371504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8372047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8372964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8373499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8374409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8374998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8375901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8376519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8377330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8378036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8378808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8379665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8380349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8381174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8381706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8382514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8383057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8383831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8384299Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.8385055Z   test_local_tensor_error (__main__.TestLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67939
2022-11-23T02:33:12.8385689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67940
2022-11-23T02:33:12.8386440Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67941
2022-11-23T02:33:12.8386972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67942
2022-11-23T02:33:12.8387924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8388473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8389392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8389954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8390860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8391428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8392301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8392915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8394476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8395172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8396107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8396660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8397551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8398095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8398991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8399531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8400278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8400827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8423647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8424228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8424943Z skip: Need at least 4 CUDA devices (2.409s)
2022-11-23T02:33:12.8425492Z   test_collect_local_shard (__main__.TestModuleHookApi) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68075
2022-11-23T02:33:12.8426276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68076
2022-11-23T02:33:12.8426829Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68077
2022-11-23T02:33:12.8427337Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68078
2022-11-23T02:33:12.8428065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8428720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8429541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8430078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8430737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8431258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8431896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8432451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8433107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8433633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8434260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8434804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8435454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8436132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8436868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8437413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8437926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8438583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8439104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8439651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8440117Z skip: Need at least 4 CUDA devices (2.409s)
2022-11-23T02:33:12.8440642Z   test_reshard_output (__main__.TestModuleHookApi) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68211
2022-11-23T02:33:12.8441233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68212
2022-11-23T02:33:12.8441764Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68213
2022-11-23T02:33:12.8442273Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68214
2022-11-23T02:33:12.8442967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8443505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8444151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8444733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8445397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8446289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8447027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8447585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8448243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8448770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8449401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8450224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8450888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8451408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8452039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8452582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8453096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8453628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8454180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8454715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8455184Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8455731Z   test_create_shard_with_no_placement (__main__.TestShardMetadata) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68347
2022-11-23T02:33:12.8456348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68348
2022-11-23T02:33:12.8456878Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68349
2022-11-23T02:33:12.8457389Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68350
2022-11-23T02:33:12.8458084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8458613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8459390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8459916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8460572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8461096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8461749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8462277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8462929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8463453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8464092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8464634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8465283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8465801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8466428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8467039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8467566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8468100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8468649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8469189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8469652Z skip: Need at least 4 CUDA devices (2.409s)
2022-11-23T02:33:12.8470185Z   test_shard_metadata_init (__main__.TestShardMetadata) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68483
2022-11-23T02:33:12.8470782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68484
2022-11-23T02:33:12.8471301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68485
2022-11-23T02:33:12.8471811Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68486
2022-11-23T02:33:12.8472508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8473033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8473683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8474207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8474838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8475331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8475969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8476783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8477475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8478270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8478908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8479546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8480203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8480723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8481358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8481899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8482416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8482948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8483490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8484030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8484498Z skip: Need at least 4 CUDA devices (2.409s)
2022-11-23T02:33:12.8485029Z   test_shard_parameter (__main__.TestShardParameter) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68619
2022-11-23T02:33:12.8485629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68620
2022-11-23T02:33:12.8486462Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68621
2022-11-23T02:33:12.8487080Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68622
2022-11-23T02:33:12.8487774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8488303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8488955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8489490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8490155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8490679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8491329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8491854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8492510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8493032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8493665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8494213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8494866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8495382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8496010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8496553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8497068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8497625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8498154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8498697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8499279Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8499819Z   test_shard_parameter_errors (__main__.TestShardParameter) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68755
2022-11-23T02:33:12.8500432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68756
2022-11-23T02:33:12.8500957Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68757
2022-11-23T02:33:12.8501479Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68758
2022-11-23T02:33:12.8502162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8502695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8503347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8503885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8504540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8505062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8505711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8506239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8506950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8507463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8508085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8508609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8509268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8509787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8510412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8510949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8511461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8512010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8512536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8513075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8513545Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:33:12.8514062Z   test_shard_tensor (__main__.TestShardTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68891
2022-11-23T02:33:12.8514648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68892
2022-11-23T02:33:12.8515166Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 68893
2022-11-23T02:33:12.8515690Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 68894
2022-11-23T02:33:12.8516365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8516895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8517539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8518048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8518779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8519321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8519990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8520516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8521174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8521700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8522351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8522875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8523533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8524056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8524686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8525229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8526076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8526723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8527261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8527801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8528272Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8528800Z   test_shard_tensor_errors (__main__.TestShardTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69027
2022-11-23T02:33:12.8529393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69028
2022-11-23T02:33:12.8529915Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69029
2022-11-23T02:33:12.8530435Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69030
2022-11-23T02:33:12.8531116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8531643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8532299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8532829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8533484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8534016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8534652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8535175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8535821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8536341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8536977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8537503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8538159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8538768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8539400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8539934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8540452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8541000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8541530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8542067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8542533Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8543068Z   test_cleanup (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69163
2022-11-23T02:33:12.8543680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69164
2022-11-23T02:33:12.8544206Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69165
2022-11-23T02:33:12.8544775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69166
2022-11-23T02:33:12.8545447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8545974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8546685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8547249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8547887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8548418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8549061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8549586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8550240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8550762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8551414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8551939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8552594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8553118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8553748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8554289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8554804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8555353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8555879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8556420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8556886Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8557435Z   test_complete_world_size (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69299
2022-11-23T02:33:12.8558128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69300
2022-11-23T02:33:12.8558651Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69301
2022-11-23T02:33:12.8559176Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69302
2022-11-23T02:33:12.8559852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8560382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8561040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8561594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8562237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8562765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8563416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8563940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8564596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8565114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8566239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8566790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8567462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8567982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8568617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8569159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8569678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8570226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8570751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8571290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8571755Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.8572189Z   test_create_sharded_tensor_like (__main__.TestShardedTensorChunked)
2022-11-23T02:33:12.8572805Z Test tensor like methods, i.e. torch.zeros_like(...), torch.full_like, etc. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69435
2022-11-23T02:33:12.8573429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69436
2022-11-23T02:33:12.8573952Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69437
2022-11-23T02:33:12.8574460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69438
2022-11-23T02:33:12.8575146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8575677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8576337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8576874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8577529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8578138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8578775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8579318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8579973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8580494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8581127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8581669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8582320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8582847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8583474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8584013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8584526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8585059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8585589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8586187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8586672Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.8587114Z   test_create_sharded_tensor_with_full (__main__.TestShardedTensorChunked)
2022-11-23T02:33:12.8587678Z Test sharded_tensor.full(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69571
2022-11-23T02:33:12.8588247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69572
2022-11-23T02:33:12.8588753Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69573
2022-11-23T02:33:12.8589273Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69574
2022-11-23T02:33:12.8589959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8590485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8591119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8591668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8592321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8592827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8593483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8594022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8594669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8595171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8595818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8596358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8597008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8597577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8598223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8598761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8599256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8599806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8600350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8600898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8601350Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8601811Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorChunked)
2022-11-23T02:33:12.8602378Z Test sharded_tensor.ones(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69707
2022-11-23T02:33:12.8602932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69708
2022-11-23T02:33:12.8603463Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69709
2022-11-23T02:33:12.8603986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69710
2022-11-23T02:33:12.8604674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8605185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8606200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8606772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8607417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8607948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8608593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8609130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8609765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8610288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8610932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8611471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8612111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8612640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8613283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8613801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8614315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8614863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8615409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8615939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8616402Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8616860Z   test_create_sharded_tensor_with_rand (__main__.TestShardedTensorChunked)
2022-11-23T02:33:12.8617520Z Test sharded_tensor.rand(...)/randn(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69843
2022-11-23T02:33:12.8618098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69844
2022-11-23T02:33:12.8618625Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69845
2022-11-23T02:33:12.8619149Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69846
2022-11-23T02:33:12.8619828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8620361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8621016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8621546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8622202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8622729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8623376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8623899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8624548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8625068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8625773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8626306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8626964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8627487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8628118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8628660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8629175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8629721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8630252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8630795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8631264Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8631707Z   test_create_sharded_tensor_with_zeros (__main__.TestShardedTensorChunked)
2022-11-23T02:33:12.8632282Z Test sharded_tensor.zeros(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69979
2022-11-23T02:33:12.8632849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69980
2022-11-23T02:33:12.8633373Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69981
2022-11-23T02:33:12.8633879Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69982
2022-11-23T02:33:12.8634568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8635102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8635741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8636287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8636944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8637552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8638186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8638732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8639380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8639899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8640529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8641067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8641716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8642224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8642867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8643405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8643920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8644495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8645096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8645654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8646336Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8646766Z   test_gather_even (__main__.TestShardedTensorChunked)
2022-11-23T02:33:12.8647363Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70115
2022-11-23T02:33:12.8647976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70116
2022-11-23T02:33:12.8648485Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70117
2022-11-23T02:33:12.8649006Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70118
2022-11-23T02:33:12.8649700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8650218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8650864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8651383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8652029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8652564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8653231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8653773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8654430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8654939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8655585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8656126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8656764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8657395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8658045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8658586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8659081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8659630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8660177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8660697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8661167Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8661652Z   test_gather_uneven (__main__.TestShardedTensorChunked)
2022-11-23T02:33:12.8662261Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70251
2022-11-23T02:33:12.8662858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70252
2022-11-23T02:33:12.8663386Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70253
2022-11-23T02:33:12.8663911Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70254
2022-11-23T02:33:12.8664585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8665197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8665871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8666419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8667065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8667589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8668235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8668776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8669406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8669931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8670569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8671088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8671739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8672265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8672906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8673429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8673940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8674493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8675022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8675568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8676029Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8676605Z   test_insufficient_sharding_dims (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70387
2022-11-23T02:33:12.8677292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70388
2022-11-23T02:33:12.8677819Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70389
2022-11-23T02:33:12.8678342Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70390
2022-11-23T02:33:12.8679036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8679545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8680196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8680739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8681375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8681903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8682550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8683090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8683724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8684245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8684944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8685476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8686498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8687026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8687666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8688186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8688697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8689243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8689788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8690317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8690779Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8691345Z   test_invalid_pg_rpc_ranks (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70523
2022-11-23T02:33:12.8691949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70524
2022-11-23T02:33:12.8692476Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70525
2022-11-23T02:33:12.8692998Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70526
2022-11-23T02:33:12.8693693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8694207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8694862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8695412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8696050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8696671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8697318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8697866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8698498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8699035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8699694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8700250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8700893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8701424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8702070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8702600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8703123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8703675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8704217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8704825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8705320Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8705890Z   test_invalid_sharding (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70659
2022-11-23T02:33:12.8706509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70660
2022-11-23T02:33:12.8707022Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70661
2022-11-23T02:33:12.8707547Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70662
2022-11-23T02:33:12.8708237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8708744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8709384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8709926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8710588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8711120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8711799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8712344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8714698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8715227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8715912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8716459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8717116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8717625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8718420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8718964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8719460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8720012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8720543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8721092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8721547Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:33:12.8722119Z   test_load_state_dict_errors (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70795
2022-11-23T02:33:12.8722744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70796
2022-11-23T02:33:12.8723274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70797
2022-11-23T02:33:12.8723809Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70798
2022-11-23T02:33:12.8724496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8725041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8726082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8726741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8727423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8727950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8728594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8729144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8729801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8730329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8730957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8731504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8732161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8732690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8733322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8733891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8734428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8734995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8735527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8736074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8736552Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8737114Z   test_multiple_local_shards (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70931
2022-11-23T02:33:12.8737741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70932
2022-11-23T02:33:12.8738366Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70933
2022-11-23T02:33:12.8738894Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70934
2022-11-23T02:33:12.8739570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8740104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8740757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8741318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8741982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8742537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8743205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8743783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8744469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8745003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8745681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8746233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8746941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8747488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8748150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8748696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8749194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8749736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8750280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8750827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8751284Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8751838Z   test_new_group (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71067
2022-11-23T02:33:12.8752444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71068
2022-11-23T02:33:12.8752956Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71069
2022-11-23T02:33:12.8753482Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71070
2022-11-23T02:33:12.8754171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8754700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8755342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8755888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8756553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8757057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8757706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8758342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8758996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8759502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8760143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8760680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8761337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8761841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8762487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8763033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8763533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8764076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8764618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8765164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8765614Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.8766525Z   test_partial_world_size (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71203
2022-11-23T02:33:12.8767156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71204
2022-11-23T02:33:12.8767664Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71205
2022-11-23T02:33:12.8768196Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71206
2022-11-23T02:33:12.8768887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8769412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8770047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8770594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8771247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8771767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8772392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8772932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8773586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8774089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8774728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8775268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8775929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8776434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8777077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8777614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8778200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8778753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8779287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8779835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8780287Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.8780863Z   test_sharded_tensor_metadata (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71339
2022-11-23T02:33:12.8781487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71340
2022-11-23T02:33:12.8781999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71341
2022-11-23T02:33:12.8782526Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71342
2022-11-23T02:33:12.8783222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8783750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8784385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8784930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8785645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8786183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8786816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8787358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8788020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8788516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8789161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8789701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8790355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8790865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8791509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8792046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8792542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8793100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8793637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8794173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8794627Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8795192Z   test_sharded_tensor_sizes (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71475
2022-11-23T02:33:12.8795813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71476
2022-11-23T02:33:12.8796341Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71477
2022-11-23T02:33:12.8796851Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71478
2022-11-23T02:33:12.8797610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8798139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8798774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8799319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8799981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8800513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8801154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8801695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8802334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8802843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8803489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8804024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8804665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8805171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8806073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8806633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8807147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8807687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8808223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8808771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8809223Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:33:12.8809785Z   test_sharding_columns (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71611
2022-11-23T02:33:12.8810400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71612
2022-11-23T02:33:12.8810932Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71613
2022-11-23T02:33:12.8811440Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71614
2022-11-23T02:33:12.8812127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8812664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8813299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8813847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8814501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8815022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8815659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8816205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8816856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8817469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8818104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8818643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8819292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8819791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8820439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8820981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8821492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8822020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8822562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8823103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8823553Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8824105Z   test_state_dict (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71747
2022-11-23T02:33:12.8824707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71748
2022-11-23T02:33:12.8825294Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71749
2022-11-23T02:33:12.8825815Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71750
2022-11-23T02:33:12.8826504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8827034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8827666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8828214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8828865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8829385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8830021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8830566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8831217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8831737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8832368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8832908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8833560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8834060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8834706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8835243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8835754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8836286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8836900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8837444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8837897Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8838464Z   test_state_dict_new_group (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71883
2022-11-23T02:33:12.8839078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71884
2022-11-23T02:33:12.8839605Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71885
2022-11-23T02:33:12.8840121Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71886
2022-11-23T02:33:12.8840813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8841342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8841982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8842530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8843190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8843714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8844389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8845007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8845857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8846387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8847030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8847570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8848219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8848716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8849364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8849907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8850423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8850957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8851492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8852040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8852492Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8853072Z   test_state_dict_no_sharded_tensors (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72019
2022-11-23T02:33:12.8853694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72020
2022-11-23T02:33:12.8854219Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72021
2022-11-23T02:33:12.8854730Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72022
2022-11-23T02:33:12.8855420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8855948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8856703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8857232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8857887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8858406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8859033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8859581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8860237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8860757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8861388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8861932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8862583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8863084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8863732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8864267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8864854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8865398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8865929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8866478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8866946Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8867486Z   test_custom_op (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72155
2022-11-23T02:33:12.8868097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72156
2022-11-23T02:33:12.8868619Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72157
2022-11-23T02:33:12.8869130Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72158
2022-11-23T02:33:12.8869816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8870340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8870987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8871522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8872180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8872700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8873322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8873863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8874517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8875040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8875669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8876285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8876944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8877464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8878086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8878624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8879137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8879670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8880208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8880751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8881231Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.8881786Z   test_custom_op_errors (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72291
2022-11-23T02:33:12.8882402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72292
2022-11-23T02:33:12.8882928Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72293
2022-11-23T02:33:12.8883435Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72294
2022-11-23T02:33:12.8884187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8884731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8885383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8886093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8886754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8887271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8887898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8888440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8889098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8889613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8890240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8890793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8891443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8891963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8892593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8893135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8893644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8894183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8894728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8895265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8895852Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8896411Z   test_custom_op_override (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72427
2022-11-23T02:33:12.8897029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72428
2022-11-23T02:33:12.8897556Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72429
2022-11-23T02:33:12.8898059Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72430
2022-11-23T02:33:12.8898766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8899293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8899943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8900474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8901133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8901652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8902297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8902820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8903540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8904068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8904695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8905232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8905888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8906408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8907032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8907568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8908080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8908617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8909164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8909701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8910166Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:33:12.8910628Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorEnumerable)
2022-11-23T02:33:12.8911207Z Test sharded_tensor.ones(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72563
2022-11-23T02:33:12.8911769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72564
2022-11-23T02:33:12.8912274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72565
2022-11-23T02:33:12.8912797Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72566
2022-11-23T02:33:12.8913485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8914010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8914644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8915271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8915927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8916453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8917085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8917631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8918287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8918793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8919446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8919992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8920660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8921162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8921816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8922355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8922880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8923476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8924029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8924577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8925039Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8925480Z   test_gather_even (__main__.TestShardedTensorEnumerable)
2022-11-23T02:33:12.8926273Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72699
2022-11-23T02:33:12.8926891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72700
2022-11-23T02:33:12.8927401Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72701
2022-11-23T02:33:12.8927925Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72702
2022-11-23T02:33:12.8928623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8929134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8929797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8930354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8931017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8931524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8932171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8932718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8933376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8933886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8934536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8935082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8935836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8936361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8937011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8937555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8938055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8938612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8939159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8939702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8940159Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.8940599Z   test_gather_uneven (__main__.TestShardedTensorEnumerable)
2022-11-23T02:33:12.8941204Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72835
2022-11-23T02:33:12.8941800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72836
2022-11-23T02:33:12.8942335Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72837
2022-11-23T02:33:12.8942861Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72838
2022-11-23T02:33:12.8943639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8944166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8944862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8945392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8946031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8946581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8947249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8947793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8948439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8948974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8949622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8950178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8950824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8951350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8951997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8952523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8953048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8953606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8954148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8954663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8955211Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:33:12.8955787Z   test_grid_sharding (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72971
2022-11-23T02:33:12.8956411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72972
2022-11-23T02:33:12.8956919Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72973
2022-11-23T02:33:12.8957446Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72974
2022-11-23T02:33:12.8958145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8958657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8959325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8959881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8960550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8961057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8961713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8962259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8962987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8963502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8964160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8964707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8965352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8966053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8966711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8967257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8967753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8968311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8968853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8969373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8969857Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.8970447Z   test_multiple_local_shards (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73107
2022-11-23T02:33:12.8971080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73108
2022-11-23T02:33:12.8971591Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73109
2022-11-23T02:33:12.8972120Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73110
2022-11-23T02:33:12.8972819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8973355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8973991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8974540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8975315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8975825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8976485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8977035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8977688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8978203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8978858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8979407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8980066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8980577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8981229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8981781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8982275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8982904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8983461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8984018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8984467Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.8985042Z   test_new_group (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73243
2022-11-23T02:33:12.8985658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73244
2022-11-23T02:33:12.8986191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73245
2022-11-23T02:33:12.8986696Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73246
2022-11-23T02:33:12.8987393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8987928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8988570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8989127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8989785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8990316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8990948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8991499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8992153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8992688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8993324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8993871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8994532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.8995110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.8995768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.8996319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.8996838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.8997372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.8997917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.8998461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.8998909Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.8999491Z   test_partial_world_size (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73379
2022-11-23T02:33:12.9000124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73380
2022-11-23T02:33:12.9000659Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73381
2022-11-23T02:33:12.9001163Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73382
2022-11-23T02:33:12.9001857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9002394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9003112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9003656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9004314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9004848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9005484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9006277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9006942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9007470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9008109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9008661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9009320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9009860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9010487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9011037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9011555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9012090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9012640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9013188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9013657Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.9014278Z   test_sharded_tensor_device (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73515
2022-11-23T02:33:12.9015018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73516
2022-11-23T02:33:12.9015546Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73517
2022-11-23T02:33:12.9016081Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73518
2022-11-23T02:33:12.9016762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9017291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9017954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9018488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9019152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9019688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9020342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9020874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9021530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9022063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9022801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9023372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9024038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9024566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9025202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9025754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9026267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9026823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9027348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9027897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9028371Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.9028944Z   test_sharded_tensor_metadata (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73651
2022-11-23T02:33:12.9029589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73652
2022-11-23T02:33:12.9030118Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73653
2022-11-23T02:33:12.9030646Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73654
2022-11-23T02:33:12.9031313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9031844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9032505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9033053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9033692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9034219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9034956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9035470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9036129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9036678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9037345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9037882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9038546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9039077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9039739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9040265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9040783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9041335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9041851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9042464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9042942Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.9043524Z   test_sharded_tensor_to_cpu (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73787
2022-11-23T02:33:12.9044139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73788
2022-11-23T02:33:12.9044720Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73789
2022-11-23T02:33:12.9045254Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73790
2022-11-23T02:33:12.9046120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9046656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9047321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9047879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9048523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9049047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9049700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9050256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9050892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9051422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9052079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9052610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9053284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9053814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9054465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9055093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9055607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9056160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9056705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9057248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9057701Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.9058278Z   test_sharded_tensor_to_cuda (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73923
2022-11-23T02:33:12.9058910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73924
2022-11-23T02:33:12.9059426Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73925
2022-11-23T02:33:12.9059940Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73926
2022-11-23T02:33:12.9060625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9061152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9061787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9062450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9063127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9063648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9064279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9064828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9065487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9065993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9066641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9067182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9067837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9068340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9068979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9069517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9070029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9070561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9071092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9071635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9072082Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:33:12.9072660Z   test_sharded_tensor_to_test (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74059
2022-11-23T02:33:12.9073284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74060
2022-11-23T02:33:12.9073808Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74061
2022-11-23T02:33:12.9074391Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74062
2022-11-23T02:33:12.9075075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9075595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9076226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9076771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9077426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9077948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9078580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9079128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9079774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9080286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9080909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9081442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9082154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9082667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9083310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9083841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9084614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9085148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9085863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9086423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9086875Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.9087445Z   test_uneven_shards (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74195
2022-11-23T02:33:12.9088067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74196
2022-11-23T02:33:12.9088589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74197
2022-11-23T02:33:12.9089105Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74198
2022-11-23T02:33:12.9089801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9090323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9090971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9091499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9092156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9092672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9093304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9093844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9094603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9095118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9095746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9096282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9096923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9097440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9098064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9098599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9099111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9099638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9100173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9100716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9101184Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.9101801Z   test_with_rpc_names (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74331
2022-11-23T02:33:12.9102431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74332
2022-11-23T02:33:12.9102957Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74333
2022-11-23T02:33:12.9103463Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74334
2022-11-23T02:33:12.9104161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9104683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9105332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9105860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9106518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9107042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9107673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9108213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9108869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9109382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9110005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9110547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9111201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9111727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9112354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9112890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9113402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9114007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9114542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9115084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9115553Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:33:12.9116736Z   test_init_from_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78068 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-11-23T02:33:12.9117854Z   test_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74467
2022-11-23T02:33:12.9118524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74468
2022-11-23T02:33:12.9119052Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74469
2022-11-23T02:33:12.9119575Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74470
2022-11-23T02:33:12.9120245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9120768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9121482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9122045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9122686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9123208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9123849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9124377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9125031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9125548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9126401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9126927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9127577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9128099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9128729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9129265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9129772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9130313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9130841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9131382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9131846Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.9132469Z   test_init_from_local_shards_and_global_metadata_invalid_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74603
2022-11-23T02:33:12.9133260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74604
2022-11-23T02:33:12.9133787Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74605
2022-11-23T02:33:12.9134311Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74606
2022-11-23T02:33:12.9134987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9135511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9136166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9136710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9137349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9137873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9138518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9139041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9139697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9140211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9140932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9141472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9142129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9142645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9143304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9143825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9144379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9144936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9145463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9146004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9146467Z skip: Need at least 4 CUDA devices (2.412s)
2022-11-23T02:33:12.9147080Z   test_init_from_local_shards_invalid_local_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74739
2022-11-23T02:33:12.9147729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74740
2022-11-23T02:33:12.9148251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74741
2022-11-23T02:33:12.9148773Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74742
2022-11-23T02:33:12.9149442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9149966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9150620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9151167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9151808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9152410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9153058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9153601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9154236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9154756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9155402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9155925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9156579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9157096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9157747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9158266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9158775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9159321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9159846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9160450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9160923Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.9161539Z   test_init_from_local_shards_invalid_pin_memory (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74875
2022-11-23T02:33:12.9162188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74876
2022-11-23T02:33:12.9162709Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74877
2022-11-23T02:33:12.9163230Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74878
2022-11-23T02:33:12.9163923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9164431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9165087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9165628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9166517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9167036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9167689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9168228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9168859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9169372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9170016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9170544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9171202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9171716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9172484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9173004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9173512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9174058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9174603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9175137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9175693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:12.9176264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:12.9176819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2022-11-23T02:33:12.9177558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:33:12.9178166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2022-11-23T02:33:12.9178892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:33:12.9179642Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:33:12.9180472Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2022-11-23T02:33:12.9181001Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:33:12.9181627Z   test_init_from_local_shards_invalid_property_cross_ranks (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75023
2022-11-23T02:33:12.9182296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75024
2022-11-23T02:33:12.9182826Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75025
2022-11-23T02:33:12.9183349Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75026
2022-11-23T02:33:12.9184020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9184547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9185205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9185751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9186391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9186925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9187566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9188088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9188720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9189263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9189928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9190450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9191107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9191701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9192349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9192871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9193382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9193931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9194462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9195008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9195468Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.9196080Z   test_init_from_local_shards_invalid_shards_gaps (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75159
2022-11-23T02:33:12.9196729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75160
2022-11-23T02:33:12.9197256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75161
2022-11-23T02:33:12.9197782Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75162
2022-11-23T02:33:12.9198475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9198988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9199695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9200245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9200886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9201410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9202059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9202599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9203231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9203751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9204398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9204922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9205571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9206321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9206980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9207502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9208016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9208564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9209111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9209638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9210099Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.9210717Z   test_init_from_local_shards_invalid_shards_overlap (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75295
2022-11-23T02:33:12.9211473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75296
2022-11-23T02:33:12.9211999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75297
2022-11-23T02:33:12.9212514Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75298
2022-11-23T02:33:12.9213203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9213713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9214356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9214878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9215513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9216065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9216741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9217316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9217982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9218494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9219204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9219759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9220397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9220917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9221565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9222099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9222593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9223142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9223685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9224214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9224682Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.9225283Z   test_init_from_local_shards_new_group (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75431
2022-11-23T02:33:12.9225938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75432
2022-11-23T02:33:12.9226448Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75433
2022-11-23T02:33:12.9226967Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75434
2022-11-23T02:33:12.9227652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9228175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9228811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9229353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9230000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9230503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9231231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9231764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9232417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9232917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9233559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9234096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9234733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9235249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9235899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9236430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9236923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9237472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9238008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9238605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9239065Z skip: Need at least 4 CUDA devices (2.510s)
2022-11-23T02:33:12.9239644Z   test_local_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75567
2022-11-23T02:33:12.9240265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75568
2022-11-23T02:33:12.9240778Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75569
2022-11-23T02:33:12.9241293Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75570
2022-11-23T02:33:12.9241983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9242504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9243134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9243683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9244333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9244879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9245530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9246247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9246908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9247405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9248046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9248590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9249240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9249743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9250386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9251029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9251524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9252070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9252616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9253156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9253610Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.9254232Z   test_st_base_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75703
2022-11-23T02:33:12.9254898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75704
2022-11-23T02:33:12.9255410Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75705
2022-11-23T02:33:12.9255933Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75706
2022-11-23T02:33:12.9256616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9257137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9257767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9258381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9259048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9259567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9260197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9260739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9261389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9261888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9262527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9263064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9263711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9264209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9264845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9265379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9265869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9266416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9266950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9267488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9267944Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:33:12.9268532Z   test_init_from_local_tensor (__main__.TestShardedTensorFromLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75839
2022-11-23T02:33:12.9269166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75840
2022-11-23T02:33:12.9269765Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75841
2022-11-23T02:33:12.9270270Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75842
2022-11-23T02:33:12.9270952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9271475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9272106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9272654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9273306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9273823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9274448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9274996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9275646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9276142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9276782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9277318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9278028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9278542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9279188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9279723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9280212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9280758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9281297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9281836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9282284Z skip: Need at least 4 CUDA devices (2.409s)
2022-11-23T02:33:12.9282675Z   test_init_from_local_tensor_errors (__main__.TestShardedTensorFromLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75975
2022-11-23T02:33:12.9282933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75976
2022-11-23T02:33:12.9283186Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75977
2022-11-23T02:33:12.9283442Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75978
2022-11-23T02:33:12.9283861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9284074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9284491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9284703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9285111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9285326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9285916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9286254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9286674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9286885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9287291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9287517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9287907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:12.9288120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:12.9288529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:12.9288757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:12.9289023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:12.9289286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:12.9289549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:33:12.9289805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:33:12.9289976Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:33:12.9290305Z   test_serialize_and_deserialize (__main__.TestShardedTensorMetadata) ... ok (0.053s)
2022-11-23T02:33:12.9290331Z 
2022-11-23T02:33:12.9290653Z ----------------------------------------------------------------------
2022-11-23T02:33:12.9290810Z Ran 64 tests in 149.607s
2022-11-23T02:33:12.9290830Z 
2022-11-23T02:33:12.9290977Z OK (skipped=62)
2022-11-23T02:33:12.9290996Z 
2022-11-23T02:33:12.9291162Z Generating XML reports...
2022-11-23T02:33:12.9291718Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20221123023042.xml
2022-11-23T02:33:12.9292262Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20221123023042.xml
2022-11-23T02:33:12.9292764Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20221123023042.xml
2022-11-23T02:33:12.9293253Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20221123023042.xml
2022-11-23T02:33:12.9293745Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20221123023042.xml
2022-11-23T02:33:12.9294251Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20221123023042.xml
2022-11-23T02:33:12.9294749Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20221123023042.xml
2022-11-23T02:33:12.9295279Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20221123023042.xml
2022-11-23T02:33:12.9295814Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20221123023042.xml
2022-11-23T02:33:12.9296361Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20221123023042.xml
2022-11-23T02:33:12.9296922Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20221123023042.xml
2022-11-23T02:33:12.9297579Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20221123023042.xml
2022-11-23T02:33:12.9297600Z 
2022-11-23T02:33:12.9298089Z ##[endgroup]
2022-11-23T02:33:12.9313107Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_f5ukg2fs)
2022-11-23T02:33:12.9313141Z 
2022-11-23T02:33:12.9313569Z Running distributed/fsdp/test_fsdp_use_orig_params ... [2022-11-23 02:33:12.836764]
2022-11-23T02:33:12.9314215Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:33:12.837039]
2022-11-23T02:36:01.3878351Z 
2022-11-23T02:36:01.3879142Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_use_orig_params
2022-11-23T02:36:01.3880212Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_bobbooql)
2022-11-23T02:36:01.3880655Z 
2022-11-23T02:36:01.3880809Z Running tests...
2022-11-23T02:36:01.3883034Z ----------------------------------------------------------------------
2022-11-23T02:36:01.3883866Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params
2022-11-23T02:36:01.3884449Z   test_named_parameters_in_forward (__main__.TestFSDPUseOrigParamsFQNs)
2022-11-23T02:36:01.3886205Z Tests that calling ``named_parameters()`` during forward returns FQNs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:36:01.3886838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76146
2022-11-23T02:36:01.3887369Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76147
2022-11-23T02:36:01.3888113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.3888627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.3889285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.3892137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.3893214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.3893764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.3894435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.3895000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.3895687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.3896535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.3897305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.3898068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.3898685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.3899356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.3899794Z dist init r=1, world=2
2022-11-23T02:36:01.3900128Z dist init r=0, world=2
2022-11-23T02:36:01.3900452Z ok (5.369s)
2022-11-23T02:36:01.3900851Z   test_param_and_buffer_names (__main__.TestFSDPUseOrigParamsFQNs)
2022-11-23T02:36:01.3901478Z Tests that, for ``use_orig_params=True``, the parameter and buffer ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76225
2022-11-23T02:36:01.3902322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76226
2022-11-23T02:36:01.3903021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.3903564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.3904226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.3904782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.3905440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.3905976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.3906632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.3907197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.3907709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.3908300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.3909040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.3909809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.3910458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.3911022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.3912379Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:36:01.3913326Z   warnings.warn(
2022-11-23T02:36:01.3914571Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:36:01.3915434Z   warnings.warn(
2022-11-23T02:36:01.3915741Z dist init r=1, world=2
2022-11-23T02:36:01.3916071Z dist init r=0, world=2
2022-11-23T02:36:01.3916384Z ok (3.309s)
2022-11-23T02:36:01.3916883Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:36:01.3917598Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76304
2022-11-23T02:36:01.3918213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76305
2022-11-23T02:36:01.3918891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.3919428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.3920069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.3920591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.3921311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.3921858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.3922518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.3923061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.3923576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.3924154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.3924892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.3925644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.3926903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.3927455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.3928011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3928561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3929001Z dist init r=0, world=2
2022-11-23T02:36:01.3929327Z dist init r=1, world=2
2022-11-23T02:36:01.3929626Z ok (4.511s)
2022-11-23T02:36:01.3930256Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:36:01.3930990Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76387
2022-11-23T02:36:01.3931610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76388
2022-11-23T02:36:01.3932349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.3932879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.3933532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.3934061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.3934722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.3935246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.3935897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.3936424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.3936957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.3937534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.3938269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.3939021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.3939623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.3940170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.3940705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3941268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3941826Z dist init r=0, world=2
2022-11-23T02:36:01.3942154Z dist init r=1, world=2
2022-11-23T02:36:01.3942455Z ok (4.611s)
2022-11-23T02:36:01.3942974Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:36:01.3943699Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76470
2022-11-23T02:36:01.3944301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76471
2022-11-23T02:36:01.3945005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.3945537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.3946187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.3946720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.3947372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.3947892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.3948523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.3949063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.3949651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.3950234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.3950959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.3951726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.3952333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.3952878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.3953419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3953983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3954424Z dist init r=0, world=2
2022-11-23T02:36:01.3954741Z dist init r=1, world=2
2022-11-23T02:36:01.3955060Z ok (4.611s)
2022-11-23T02:36:01.3955561Z   test_diff_hyperparams_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:36:01.3956250Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76553
2022-11-23T02:36:01.3956875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76554
2022-11-23T02:36:01.3957568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.3958094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.3958728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.3959274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.3959935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.3960458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.3961088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.3961691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.3962220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.3962776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.3963516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.3964274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.3964872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.3965405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.3966388Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3966969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3967525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3968057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3968607Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3969154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3969782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3970351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3970896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3971439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3971973Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3972515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3973056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3973587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3974136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3974686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3975229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3975759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3976298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3976843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3977384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3977916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3978458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3978996Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3979526Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3980067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3980601Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3981143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3981754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3982292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3982832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3983354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3983893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3984439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3984981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3985507Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3986052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3986591Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3987115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3987658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3988193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3988787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3989325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3989864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3990404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3990932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3991475Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3992015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3993386Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:36:01.3994254Z   warnings.warn(
2022-11-23T02:36:01.3995476Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:36:01.3996325Z   warnings.warn(
2022-11-23T02:36:01.3996770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3997323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3997862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3998419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3998965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.3999595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4000128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4000671Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4001213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4001742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4002283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4002826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4003372Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4003907Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4004453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4004990Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4005518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4006776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4007323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4007959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4008504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4009044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4009586Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4010134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4010663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4011205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4011745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4012272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4012827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4013378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4013923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4014453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4015000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4015543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4016069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4016618Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4017161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4017701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4018232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4018775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4019401Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4019928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4020472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4021011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4021550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4022078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4022621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4023162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4023576Z dist init r=0, world=2
2022-11-23T02:36:01.4023908Z dist init r=1, world=2
2022-11-23T02:36:01.4024232Z ok (32.354s)
2022-11-23T02:36:01.4024732Z   test_diff_hyperparams_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:36:01.4025424Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76636
2022-11-23T02:36:01.4026044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76637
2022-11-23T02:36:01.4026753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4027323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4027994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4028539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4029191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4029698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4030347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4030884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4031417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.4031978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.4032773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4033543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4034127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.4034679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.4035227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4035783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4036323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4036881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4037971Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4039383Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4040699Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4042000Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4043319Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4044639Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4046481Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4047834Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4049139Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4050443Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4051254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4051803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4052359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4052915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4053447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4053998Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4054548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4055094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4055627Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4056266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4056811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4057357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4057889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4058437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4058988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4059518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4060062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4060604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4061153Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4061684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4062226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4062765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4063292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4063895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4064986Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4066312Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4067630Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4068946Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4070247Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4071554Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4072859Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4074235Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4075537Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4076847Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4077657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4078201Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4078755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4079305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4079849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4080443Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4081006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4081549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4082076Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4082632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4083702Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4085017Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4086801Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4088127Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4089424Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4090720Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4092146Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4093452Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4094745Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4096048Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4096851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4097463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4098032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4098580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4099644Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4100960Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4102252Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4103556Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4104870Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4106177Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4107477Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4108851Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4110153Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4111448Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4112241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4112804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4113356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4113963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4114508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4115059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4116409Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:36:01.4117274Z   warnings.warn(
2022-11-23T02:36:01.4118487Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:36:01.4119345Z   warnings.warn(
2022-11-23T02:36:01.4119802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4120359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4120915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4121446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4121992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4122542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4123071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4123616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4124157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4124784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4125315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4126284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4126846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4127375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4127921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4128467Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4129011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4129540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4130637Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4132066Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4133452Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4134770Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4136078Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4137364Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4138678Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4139980Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4141279Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4142668Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4143470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4144033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4144577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4145126Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4145675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4146209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4146770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4147314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4147857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4148387Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4148930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4149524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4150063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4150610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4151161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4151701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4152234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4152780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4153321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4153866Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4154402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4154945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4155487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4156020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4157094Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4158411Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4159718Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4161109Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4162422Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4163733Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4165024Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4166825Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4168168Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4169480Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:36:01.4170287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4170849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4171394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4171948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4172491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4173043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4173463Z dist init r=1, world=2
2022-11-23T02:36:01.4173785Z dist init r=0, world=2
2022-11-23T02:36:01.4174101Z ok (32.854s)
2022-11-23T02:36:01.4174586Z   test_diff_hyperparams_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:36:01.4175297Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76719
2022-11-23T02:36:01.4175920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76720
2022-11-23T02:36:01.4176598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4177132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4177784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4178423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4179068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4179592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4180245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4180784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4181302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.4181874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.4182614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4183371Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4183968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.4184519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.4185073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4185621Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4186229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4186793Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4187342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4187882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4188429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4188972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4189499Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4190040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4190636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4191190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4191723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4192267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4192820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4193347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4193894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4194437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4194983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4195517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4196062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4196602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4197129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4197747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4198285Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4198827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4199352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4199892Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4200437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4200962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4201508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4202057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4202597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4203123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4203664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4204203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4204744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4205327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4206293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4206852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4207389Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4207942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4208494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4209039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4209570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4210122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4210666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4211195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4212561Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:36:01.4213424Z   warnings.warn(
2022-11-23T02:36:01.4214671Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:36:01.4215520Z   warnings.warn(
2022-11-23T02:36:01.4216057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4216616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4217162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4217708Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4218236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4218782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4219330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4219858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4220413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4220959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4221502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4222032Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4222573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4223119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4223763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4224311Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4224856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4225399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4225933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4226476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4227023Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4227566Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4228098Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4228646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4229190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4229720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4230269Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4230814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4231351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4231878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4232465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4233008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4233537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4234081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4234623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4235227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4235754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4236293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4236835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4237376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4237904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4238449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4238990Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4239516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4240069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4240611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4241149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4241675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4242106Z dist init r=0, world=2
2022-11-23T02:36:01.4242433Z dist init r=1, world=2
2022-11-23T02:36:01.4242735Z ok (31.952s)
2022-11-23T02:36:01.4243241Z   test_diff_trainability (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:36:01.4243918Z Tests FSDP parity with DDP when using multiple parameter groups and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76802
2022-11-23T02:36:01.4244520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76803
2022-11-23T02:36:01.4245227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4246079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4246892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4247426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4248080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4248605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4249258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4249784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4250311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.4250893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.4251614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4252386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4252990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.4253539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.4253960Z dist init r=0, world=2
2022-11-23T02:36:01.4254289Z dist init r=1, world=2
2022-11-23T02:36:01.4254609Z ok (7.415s)
2022-11-23T02:36:01.4255044Z   test_multiple_optimizers (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:36:01.4255828Z Tests using two optimizers where only one sets gradients to ``None``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76885
2022-11-23T02:36:01.4256438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76886
2022-11-23T02:36:01.4257132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4257644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4258294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4258844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4259483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4260003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4260654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4261204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4261716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.4262289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.4263029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4263866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4264461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.4265010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.4266122Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:36:01.4267734Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:36:01.4268685Z dist init r=0, world=2
2022-11-23T02:36:01.4269014Z dist init r=1, world=2
2022-11-23T02:36:01.4269313Z ok (4.311s)
2022-11-23T02:36:01.4269743Z   test_access_params_after_forward (__main__.TestFSDPUseOrigParamsParamAccess)
2022-11-23T02:36:01.4270390Z Tests that accessing the original parameters after the forward but ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76968
2022-11-23T02:36:01.4271007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76969
2022-11-23T02:36:01.4271692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4272219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4272869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4273466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4274132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4274653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4275301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4275827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4276360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.4276932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.4277652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4278426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4279029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.4279575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.4280112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4280674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4281287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4281853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4282388Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4282935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4283375Z dist init r=1, world=2
2022-11-23T02:36:01.4283685Z dist init r=0, world=2
2022-11-23T02:36:01.4284003Z ok (3.810s)
2022-11-23T02:36:01.4284461Z   test_multiple_forward_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard)
2022-11-23T02:36:01.4285099Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77051
2022-11-23T02:36:01.4285992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77052
2022-11-23T02:36:01.4286778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4287306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4287941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4288488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4289143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4289664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4290296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4290836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4291363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.4291924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.4292659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4293425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4294134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.4294665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.4295093Z dist init r=1, world=2
2022-11-23T02:36:01.4295421Z dist init r=0, world=2
2022-11-23T02:36:01.4295722Z ok (4.912s)
2022-11-23T02:36:01.4296178Z   test_multiple_forward_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard)
2022-11-23T02:36:01.4296833Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77134
2022-11-23T02:36:01.4297435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77135
2022-11-23T02:36:01.4298115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4298650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4299302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4299850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4300491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4301020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4301737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4302278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4302808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.4303380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.4304121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4304860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4305457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.4306006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.4306422Z dist init r=0, world=2
2022-11-23T02:36:01.4306755Z dist init r=1, world=2
2022-11-23T02:36:01.4307073Z ok (5.112s)
2022-11-23T02:36:01.4307554Z   test_summon_between_two_forwards_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard)
2022-11-23T02:36:01.4308205Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77217
2022-11-23T02:36:01.4308808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77218
2022-11-23T02:36:01.4309497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4310007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4310659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4311198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4311856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4312363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4313008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4313613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4314123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.4314694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.4315425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4316182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4316767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.4317315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.4317740Z dist init r=0, world=2
2022-11-23T02:36:01.4318064Z dist init r=1, world=2
2022-11-23T02:36:01.4318366Z ok (5.312s)
2022-11-23T02:36:01.4318839Z   test_summon_between_two_forwards_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard)
2022-11-23T02:36:01.4319500Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77300
2022-11-23T02:36:01.4320086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77301
2022-11-23T02:36:01.4320777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4321360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4322026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4322555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4323205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4323726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4324360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4324903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4325433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.4326333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.4327073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4327848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4328443Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.4328991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.4329408Z dist init r=0, world=2
2022-11-23T02:36:01.4329737Z dist init r=1, world=2
2022-11-23T02:36:01.4330051Z ok (5.413s)
2022-11-23T02:36:01.4330442Z   test_grad_writeback (__main__.TestFSDPUseOrigParamsWriteback)
2022-11-23T02:36:01.4331200Z Tests that changes to the original parameters' gradients are written ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77383
2022-11-23T02:36:01.4331818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77384
2022-11-23T02:36:01.4332528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4333059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4333708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4334359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4335002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4335528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4336177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4336716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4337236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.4337809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.4338540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4339298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4339897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.4340442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.4341000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4341616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4342190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4342738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4343266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4343826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4344373Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4344915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4345442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4345990Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4346532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4347080Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:36:01.4347498Z dist init r=1, world=2
2022-11-23T02:36:01.4347822Z dist init r=0, world=2
2022-11-23T02:36:01.4348136Z ok (3.810s)
2022-11-23T02:36:01.4348527Z   test_param_writeback (__main__.TestFSDPUseOrigParamsWriteback)
2022-11-23T02:36:01.4349150Z Tests that changes to the original parameters are written back. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77466
2022-11-23T02:36:01.4349759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77467
2022-11-23T02:36:01.4350437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4350963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4351620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4352167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4352811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4353333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4354045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4354590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4355106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.4355679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.4356422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4357176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4357775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.4358318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.4358756Z dist init r=0, world=2
2022-11-23T02:36:01.4359069Z dist init r=1, world=2
2022-11-23T02:36:01.4359387Z ok (3.309s)
2022-11-23T02:36:01.4359936Z   test_writeback_shape_mismatch (__main__.TestFSDPUseOrigParamsWriteback) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77545
2022-11-23T02:36:01.4360563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77546
2022-11-23T02:36:01.4361252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4361833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4362497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4363030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4363695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:36:01.4364215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:36:01.4364847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:36:01.4365387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:36:01.4366289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:36:01.4366874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:36:01.4367605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4368372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:36:01.4368974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:36:01.4369517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:36:01.4369932Z dist init r=0, world=2
2022-11-23T02:36:01.4370262Z dist init r=1, world=2
2022-11-23T02:36:01.4370578Z ok (3.210s)
2022-11-23T02:36:01.4370747Z 
2022-11-23T02:36:01.4371063Z ----------------------------------------------------------------------
2022-11-23T02:36:01.4371472Z Ran 18 tests in 166.189s
2022-11-23T02:36:01.4371673Z 
2022-11-23T02:36:01.4371806Z OK
2022-11-23T02:36:01.4371978Z 
2022-11-23T02:36:01.4372127Z Generating XML reports...
2022-11-23T02:36:01.4372852Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20221123023314.xml
2022-11-23T02:36:01.4373850Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20221123023314.xml
2022-11-23T02:36:01.4374966Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20221123023314.xml
2022-11-23T02:36:01.4375943Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20221123023314.xml
2022-11-23T02:36:01.4376919Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20221123023314.xml
2022-11-23T02:36:01.4377354Z 
2022-11-23T02:36:01.4377830Z ##[endgroup]
2022-11-23T02:36:01.4378542Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_bobbooql)
2022-11-23T02:36:01.4378961Z 
2022-11-23T02:36:01.4379276Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2022-11-23 02:36:01.388723]
2022-11-23T02:36:01.4380070Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:36:01.389033]
2022-11-23T02:40:17.2999310Z 
2022-11-23T02:40:17.2999864Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_mixed_precision
2022-11-23T02:40:17.3002764Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_p88flszp)
2022-11-23T02:40:17.3003223Z 
2022-11-23T02:40:17.3003355Z Running tests...
2022-11-23T02:40:17.3004673Z ----------------------------------------------------------------------
2022-11-23T02:40:17.3005361Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision
2022-11-23T02:40:17.3007487Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77659
2022-11-23T02:40:17.3008165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77660
2022-11-23T02:40:17.3008933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3009455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3010109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3010636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3011289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3011805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3012465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3012990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3013520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3014133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3014858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3015627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3016228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3019227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3019688Z dist init r=0, world=2
2022-11-23T02:40:17.3020044Z dist init r=1, world=2
2022-11-23T02:40:17.3020568Z ok (4.726s)
2022-11-23T02:40:17.3021246Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77743
2022-11-23T02:40:17.3022073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77744
2022-11-23T02:40:17.3022862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3023424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3024131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3024688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3025380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3025933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3026615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3027174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3027726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3028321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3029184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3030019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3030646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3031219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3032218Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3033012Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3033950Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3034636Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3035692Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3037105Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3038513Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3039921Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3041417Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3042815Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3044231Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3045652Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3047522Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3048868Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3049561Z dist init r=0, world=2
2022-11-23T02:40:17.3049878Z dist init r=1, world=2
2022-11-23T02:40:17.3050190Z ok (5.012s)
2022-11-23T02:40:17.3050819Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77827
2022-11-23T02:40:17.3051524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77828
2022-11-23T02:40:17.3052206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3052746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3053400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3053960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3054598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3055123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3055780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3056313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3056848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3057437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3058163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3058915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3059612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3060174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3061133Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3061792Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3062682Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3063342Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3064346Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3065734Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3067058Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3068352Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3069656Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3070965Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3072266Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3073555Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3074853Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3076220Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3076896Z dist init r=1, world=2
2022-11-23T02:40:17.3077207Z dist init r=0, world=2
2022-11-23T02:40:17.3077518Z ok (5.012s)
2022-11-23T02:40:17.3078161Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77911
2022-11-23T02:40:17.3078874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77912
2022-11-23T02:40:17.3079563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3080090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3080744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3081276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3081927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3082447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3083132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3083682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3084208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3084772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3085497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3086821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3087421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3087965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3088918Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3089587Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3090465Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3091124Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3092102Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3093428Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3094749Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3096180Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3097487Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3098785Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3100089Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3101454Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3102752Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3104050Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3104735Z dist init r=0, world=2
2022-11-23T02:40:17.3105067Z dist init r=1, world=2
2022-11-23T02:40:17.3105382Z ok (5.013s)
2022-11-23T02:40:17.3105973Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77995
2022-11-23T02:40:17.3106669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77996
2022-11-23T02:40:17.3107356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3107867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3108516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3109060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3109717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3110223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3110869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3111407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3112001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3112554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3113290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3114101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3114687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3115233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3116184Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3116854Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3117721Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3118377Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3119427Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3120756Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3122068Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3123368Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3124679Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3126384Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3127724Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3129030Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3130442Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3131745Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3132426Z dist init r=1, world=2
2022-11-23T02:40:17.3132737Z dist init r=0, world=2
2022-11-23T02:40:17.3133051Z ok (5.113s)
2022-11-23T02:40:17.3133691Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78079
2022-11-23T02:40:17.3134418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78080
2022-11-23T02:40:17.3135092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3135621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3136340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3136901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3137542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3138060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3138707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3139234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3139763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3140334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3141061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3141813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3142407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3142948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3143911Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3144553Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3145425Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3146084Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3147079Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3148451Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3149764Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3151079Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3152386Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3153740Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3155045Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3156354Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3157649Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3158950Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3159621Z dist init r=1, world=2
2022-11-23T02:40:17.3159943Z dist init r=0, world=2
2022-11-23T02:40:17.3160261Z ok (5.213s)
2022-11-23T02:40:17.3160855Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78163
2022-11-23T02:40:17.3161551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78164
2022-11-23T02:40:17.3162240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3162768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3163402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3163947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3164658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3165173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3166251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3166805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3167334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3167894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3168634Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3169397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3169998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3170526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3171473Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3172141Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3173112Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3173764Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3174769Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3176093Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3177411Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3178716Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3180033Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3181337Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3182623Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3184003Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3185308Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3186601Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3187283Z dist init r=1, world=2
2022-11-23T02:40:17.3187608Z dist init r=0, world=2
2022-11-23T02:40:17.3187905Z ok (5.014s)
2022-11-23T02:40:17.3188592Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78247
2022-11-23T02:40:17.3189323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78248
2022-11-23T02:40:17.3190013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3190522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3191181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3191723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3192363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3192884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3193532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3194073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3194583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3195147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3195880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3197104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3197688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3198235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3199192Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3199854Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3200709Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3201424Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3202424Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3203745Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3205057Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3206829Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3208238Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3209566Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3210875Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3212185Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3213477Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3214821Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3215496Z dist init r=0, world=2
2022-11-23T02:40:17.3215807Z dist init r=1, world=2
2022-11-23T02:40:17.3216121Z ok (5.212s)
2022-11-23T02:40:17.3216733Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78331
2022-11-23T02:40:17.3217408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78332
2022-11-23T02:40:17.3218179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3218707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3219357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3219886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3220538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3221059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3221689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3222233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3222763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3223332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3224045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3224811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3225409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3226006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3226959Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3227626Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3228501Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3229146Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3230128Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3231446Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3232760Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3234069Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3235368Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3236739Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3238039Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3239343Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3240628Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3241980Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3242668Z dist init r=1, world=2
2022-11-23T02:40:17.3242994Z dist init r=0, world=2
2022-11-23T02:40:17.3243305Z ok (5.012s)
2022-11-23T02:40:17.3243905Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78415
2022-11-23T02:40:17.3244609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78416
2022-11-23T02:40:17.3245296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3246167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3246850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3247397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3248045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3248548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3249192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3249733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3266208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3266880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3267656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3268432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3269042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3269604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3270740Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3271397Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3272293Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3272956Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3273970Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3275299Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3276610Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3278024Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3279356Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3280654Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3281969Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3283283Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3284595Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3286286Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3287083Z dist init r=1, world=2
2022-11-23T02:40:17.3287417Z dist init r=0, world=2
2022-11-23T02:40:17.3287734Z ok (5.012s)
2022-11-23T02:40:17.3288313Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78499
2022-11-23T02:40:17.3289005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78500
2022-11-23T02:40:17.3289715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3290250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3290890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3291441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3292100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3292634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3293270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3293809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3294343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3294908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3295719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3296505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3297110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3297650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3298611Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3299280Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3300174Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3300820Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3301825Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3303149Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3304460Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3305785Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3307154Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3308471Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3309778Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3311077Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3312437Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3313756Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3314492Z dist init r=1, world=2
2022-11-23T02:40:17.3314824Z dist init r=0, world=2
2022-11-23T02:40:17.3315127Z ok (5.112s)
2022-11-23T02:40:17.3315751Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78583
2022-11-23T02:40:17.3316466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78584
2022-11-23T02:40:17.3317165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3317676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3318377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3318935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3319578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3320103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3320750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3321289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3321805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3322389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3323124Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3323964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3324545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3325095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3326424Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3327109Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3327985Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3328649Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3329653Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3331062Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3332409Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3333708Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3335030Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3336339Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3337659Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3338970Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3340285Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3341674Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3342359Z dist init r=1, world=2
2022-11-23T02:40:17.3342674Z dist init r=0, world=2
2022-11-23T02:40:17.3342990Z ok (5.012s)
2022-11-23T02:40:17.3343583Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78667
2022-11-23T02:40:17.3344251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78668
2022-11-23T02:40:17.3344948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3345481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3346139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3346676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3347334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3347859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3348546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3349098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3349632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3350215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3350939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3351705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3352304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3352857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3353806Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3354471Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3355358Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3356013Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3357001Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3358337Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3359710Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3361027Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3362339Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3363658Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3364960Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3366723Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3368048Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3369356Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3370048Z dist init r=1, world=2
2022-11-23T02:40:17.3370382Z dist init r=0, world=2
2022-11-23T02:40:17.3370679Z ok (5.012s)
2022-11-23T02:40:17.3371303Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78751
2022-11-23T02:40:17.3372023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78752
2022-11-23T02:40:17.3372715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3373228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3373885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3374437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3375097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3375600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3376248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3376890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3377402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3377986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3378727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3379587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3380549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3381307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3382292Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3382967Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3383832Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3384499Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3385583Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3386922Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3388229Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3389563Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3390874Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3392202Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3393519Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3394828Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3396209Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3397515Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3398203Z dist init r=0, world=2
2022-11-23T02:40:17.3398516Z dist init r=1, world=2
2022-11-23T02:40:17.3398841Z ok (5.212s)
2022-11-23T02:40:17.3399437Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78835
2022-11-23T02:40:17.3400121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78836
2022-11-23T02:40:17.3400797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3401328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3402040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3402581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3403241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3403768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3404424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3404950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3405482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3406419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3407143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3407923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3408527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3409081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3410036Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3410711Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3411607Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3412261Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3413263Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3414722Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3416044Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3417359Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3418665Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3420025Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3421343Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3422655Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3423961Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3425253Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3425952Z dist init r=0, world=2
2022-11-23T02:40:17.3426285Z dist init r=1, world=2
2022-11-23T02:40:17.3426601Z ok (5.112s)
2022-11-23T02:40:17.3427200Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78919
2022-11-23T02:40:17.3427906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78920
2022-11-23T02:40:17.3428602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3429132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3429770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3430316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3431039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3431540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3432193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3432735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3433264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3433832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3434573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3435335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3435945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3436481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3437446Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3438112Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3439054Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3439702Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3440715Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3442036Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3443367Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3444697Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3446344Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3447674Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3449071Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3450383Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3451698Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3453011Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3453693Z dist init r=0, world=2
2022-11-23T02:40:17.3454029Z dist init r=1, world=2
2022-11-23T02:40:17.3454335Z ok (5.213s)
2022-11-23T02:40:17.3454932Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79003
2022-11-23T02:40:17.3455688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79004
2022-11-23T02:40:17.3456379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3456914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3457579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3458129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3458769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3459295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3459947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3460475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3461012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3461592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3462335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3463086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3463681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3464230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3465199Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3465851Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3466741Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3467462Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3468473Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3469807Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3471103Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3472424Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3473797Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3475124Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3476437Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3477761Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3479068Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3480384Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3481049Z dist init r=1, world=2
2022-11-23T02:40:17.3481385Z dist init r=0, world=2
2022-11-23T02:40:17.3481709Z ok (5.112s)
2022-11-23T02:40:17.3482326Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79087
2022-11-23T02:40:17.3483049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79088
2022-11-23T02:40:17.3483805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3484336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3484975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3485524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3486628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3487164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3487800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3488348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3488885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3489475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3490195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3490961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3491560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3492187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3493567Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3494453Z   world_indices[
2022-11-23T02:40:17.3495700Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3496568Z   world_indices[
2022-11-23T02:40:17.3496877Z dist init r=0, world=2
2022-11-23T02:40:17.3497214Z dist init r=1, world=2
2022-11-23T02:40:17.3497536Z ok (5.012s)
2022-11-23T02:40:17.3498131Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79171
2022-11-23T02:40:17.3498807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79172
2022-11-23T02:40:17.3499503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3500032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3500670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3501227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3501886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3502412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3503045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3503687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3504218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3504802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3505524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3506295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3506898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3507428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3508518Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3509842Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3511211Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3512523Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3513219Z dist init r=0, world=2
2022-11-23T02:40:17.3513534Z dist init r=1, world=2
2022-11-23T02:40:17.3513894Z ok (4.912s)
2022-11-23T02:40:17.3514526Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79255
2022-11-23T02:40:17.3515245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79256
2022-11-23T02:40:17.3515924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3516455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3517114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3517645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3518307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3518831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3519481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3520010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3520544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3521122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3521914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3522686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3523290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3523840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3525190Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3526332Z   world_indices[
2022-11-23T02:40:17.3527577Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3528439Z   world_indices[
2022-11-23T02:40:17.3528764Z dist init r=0, world=2
2022-11-23T02:40:17.3529158Z dist init r=1, world=2
2022-11-23T02:40:17.3529488Z ok (5.012s)
2022-11-23T02:40:17.3530083Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79339
2022-11-23T02:40:17.3530767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79340
2022-11-23T02:40:17.3531449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3531981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3532636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3533168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3533831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3534357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3535008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3535535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3536075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3536657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3537374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3538145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3538752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3539296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3540358Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3541771Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3543101Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3544405Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3545101Z dist init r=1, world=2
2022-11-23T02:40:17.3545437Z dist init r=0, world=2
2022-11-23T02:40:17.3545737Z ok (4.912s)
2022-11-23T02:40:17.3546360Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79423
2022-11-23T02:40:17.3547076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79424
2022-11-23T02:40:17.3547802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3548344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3548997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3549547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3550186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3550715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3551360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3551903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3552421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3553002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3553738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3554489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3555090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3555642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3557004Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3557885Z   world_indices[
2022-11-23T02:40:17.3559109Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3560027Z   world_indices[
2022-11-23T02:40:17.3560356Z dist init r=0, world=2
2022-11-23T02:40:17.3560685Z dist init r=1, world=2
2022-11-23T02:40:17.3560984Z ok (5.112s)
2022-11-23T02:40:17.3561573Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79507
2022-11-23T02:40:17.3562259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79508
2022-11-23T02:40:17.3562933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3563469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3564124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3564673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3565312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3566072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3566819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3567373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3567889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3568472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3569217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3569967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3570573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3571126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3572476Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3573431Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:40:17.3574736Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3575697Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:40:17.3576109Z dist init r=0, world=2
2022-11-23T02:40:17.3576443Z dist init r=1, world=2
2022-11-23T02:40:17.3576749Z ok (5.012s)
2022-11-23T02:40:17.3577372Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79591
2022-11-23T02:40:17.3578167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79592
2022-11-23T02:40:17.3578862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3579376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3580030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3580581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3581220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3581742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3582390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3582938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3583453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3584036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3584768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3585596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3586189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3586742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3588100Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3588976Z   world_indices[
2022-11-23T02:40:17.3590200Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3591065Z   world_indices[
2022-11-23T02:40:17.3591393Z dist init r=0, world=2
2022-11-23T02:40:17.3591723Z dist init r=1, world=2
2022-11-23T02:40:17.3592025Z ok (5.112s)
2022-11-23T02:40:17.3592611Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79675
2022-11-23T02:40:17.3593296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79676
2022-11-23T02:40:17.3593983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3594500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3595162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3595710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3596352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3596942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3597593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3598135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3598648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3599229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3599970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3600736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3601325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3601879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3603227Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3604232Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:40:17.3605567Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3606736Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:40:17.3607152Z dist init r=1, world=2
2022-11-23T02:40:17.3607482Z dist init r=0, world=2
2022-11-23T02:40:17.3607783Z ok (5.012s)
2022-11-23T02:40:17.3608428Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79759
2022-11-23T02:40:17.3609153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79760
2022-11-23T02:40:17.3609849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3610369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3611021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3611568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3612227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3612731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3613385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3613961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3614477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3615059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3615888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3616656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3617238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3617793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3618799Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3619472Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3620339Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3621001Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3622009Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3623401Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3624739Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3626060Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3627366Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3628670Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3629987Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3631296Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3632603Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3633977Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3634659Z dist init r=1, world=2
2022-11-23T02:40:17.3634980Z dist init r=0, world=2
2022-11-23T02:40:17.3635299Z ok (5.112s)
2022-11-23T02:40:17.3635916Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79843
2022-11-23T02:40:17.3636620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79844
2022-11-23T02:40:17.3637304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3637839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3638493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3639028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3639745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3640277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3640933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3641458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3641995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3642573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3643291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3644060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3644661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3645210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3646331Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3647011Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3647907Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3648567Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3649577Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3650890Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3652299Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3653616Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3654923Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3656241Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3657612Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3658926Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3660228Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3661515Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3662211Z dist init r=1, world=2
2022-11-23T02:40:17.3662546Z dist init r=0, world=2
2022-11-23T02:40:17.3662869Z ok (5.012s)
2022-11-23T02:40:17.3663497Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79927
2022-11-23T02:40:17.3664233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79928
2022-11-23T02:40:17.3664929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3665460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3666105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3666655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3667315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3667825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3668543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3669085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3669619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3670179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3670915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3671687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3672290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3672819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3673783Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3674451Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3675332Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3676024Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3677040Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3678367Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3679683Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3680989Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3682312Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3683627Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3684921Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3686471Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3687783Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3689071Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3689763Z dist init r=1, world=2
2022-11-23T02:40:17.3690095Z dist init r=0, world=2
2022-11-23T02:40:17.3690396Z ok (5.112s)
2022-11-23T02:40:17.3691009Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80011
2022-11-23T02:40:17.3691710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80012
2022-11-23T02:40:17.3692462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3693007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3693666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3694213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3694859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3695386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3696038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3696565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3697099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3697682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3698421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3699170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3699780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3700333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3701296Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3701949Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3702837Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3703498Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3704500Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3705905Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3707206Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3708535Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3709854Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3711208Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3712517Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3713812Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3715161Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3716470Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3717142Z dist init r=1, world=2
2022-11-23T02:40:17.3717482Z dist init r=0, world=2
2022-11-23T02:40:17.3717802Z ok (5.012s)
2022-11-23T02:40:17.3718448Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80095
2022-11-23T02:40:17.3719164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80096
2022-11-23T02:40:17.3719854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3720381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3721084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3721638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3722294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3722823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3723454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3724006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3724541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3725127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3726033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3726805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3727408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3727944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3728982Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3729665Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3730559Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3731218Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3732203Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3733527Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3734844Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3736151Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3737470Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3738778Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3740169Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3741452Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3742773Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3744086Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3744773Z dist init r=1, world=2
2022-11-23T02:40:17.3745110Z dist init r=0, world=2
2022-11-23T02:40:17.3745413Z ok (5.112s)
2022-11-23T02:40:17.3746073Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80179
2022-11-23T02:40:17.3746779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80180
2022-11-23T02:40:17.3747483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3747996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3748647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3749194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3749829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3750359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3751012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3751555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3752071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3752657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3753397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3754168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3754753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3755307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3756272Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3757014Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3757882Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3758543Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3759549Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3760873Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3762183Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3763551Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3764856Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3766409Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3767725Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3769034Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3770340Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3771653Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3772346Z dist init r=0, world=2
2022-11-23T02:40:17.3772658Z dist init r=1, world=2
2022-11-23T02:40:17.3772981Z ok (5.012s)
2022-11-23T02:40:17.3773626Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80263
2022-11-23T02:40:17.3774427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80264
2022-11-23T02:40:17.3775124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3775658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3776314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3776853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3777512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3778042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3778695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3779230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3779765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3780345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3781067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3781908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3782524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3783078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3784033Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3784700Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3785581Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3786241Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3787226Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3788553Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3789869Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3791186Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3792560Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3793858Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3795176Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3796483Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3797793Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3799130Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3799831Z dist init r=0, world=2
2022-11-23T02:40:17.3800164Z dist init r=1, world=2
2022-11-23T02:40:17.3800488Z ok (5.112s)
2022-11-23T02:40:17.3801081Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80347
2022-11-23T02:40:17.3801784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80348
2022-11-23T02:40:17.3802477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3803010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3803651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3804205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3804861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3805376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3806273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3806807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3807343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3807912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3808660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3809427Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3810128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3810663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3811623Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3812290Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3813186Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3813827Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3814867Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3816196Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3817569Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3818905Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3820226Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3821525Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3822843Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3824163Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3825457Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3826773Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3827519Z dist init r=0, world=2
2022-11-23T02:40:17.3827834Z dist init r=1, world=2
2022-11-23T02:40:17.3828155Z ok (5.112s)
2022-11-23T02:40:17.3828789Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80431
2022-11-23T02:40:17.3829520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80432
2022-11-23T02:40:17.3830195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3830724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3831382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3831936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3832573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3833099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3833747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3834272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3834864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3835452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3836193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3836950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3837550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3838099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3839460Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3840422Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:40:17.3841727Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3842686Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:40:17.3843094Z dist init r=1, world=2
2022-11-23T02:40:17.3843430Z dist init r=0, world=2
2022-11-23T02:40:17.3843734Z ok (5.012s)
2022-11-23T02:40:17.3844339Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80515
2022-11-23T02:40:17.3845033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80516
2022-11-23T02:40:17.3845513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3845942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3846378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3846610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3847027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3847241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3847658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3847887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3848178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3848463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3848883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3849322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3849676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3849955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3850757Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3851550Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3852327Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3853112Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3853272Z dist init r=0, world=2
2022-11-23T02:40:17.3853421Z dist init r=1, world=2
2022-11-23T02:40:17.3853563Z ok (5.112s)
2022-11-23T02:40:17.3854028Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80599
2022-11-23T02:40:17.3854270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80600
2022-11-23T02:40:17.3854688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3854905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3855329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3855633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3856040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3856257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3856671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3856901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3857172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3857456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3857897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3858341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3858612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3858879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3859991Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3860243Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:40:17.3861304Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3861551Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:40:17.3861705Z dist init r=0, world=2
2022-11-23T02:40:17.3861835Z dist init r=1, world=2
2022-11-23T02:40:17.3861976Z ok (4.914s)
2022-11-23T02:40:17.3862416Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80683
2022-11-23T02:40:17.3862676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80684
2022-11-23T02:40:17.3863089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3863308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3863731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3863961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3864353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3864573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3864988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3865215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3865501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3865860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3866303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3866740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3867010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3867261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3868055Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3868832Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3869671Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3870451Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:40:17.3870610Z dist init r=0, world=2
2022-11-23T02:40:17.3870759Z dist init r=1, world=2
2022-11-23T02:40:17.3870899Z ok (5.013s)
2022-11-23T02:40:17.3871363Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80767
2022-11-23T02:40:17.3871621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80768
2022-11-23T02:40:17.3872037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3872253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3872656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3872891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3873303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3873517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3873930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3874158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3874443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3874726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3875168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3875589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3875914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3876180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3877259Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3877412Z   world_indices[
2022-11-23T02:40:17.3878479Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3878631Z   world_indices[
2022-11-23T02:40:17.3878781Z dist init r=1, world=2
2022-11-23T02:40:17.3878931Z dist init r=0, world=2
2022-11-23T02:40:17.3879072Z ok (5.012s)
2022-11-23T02:40:17.3879487Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80851
2022-11-23T02:40:17.3879798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80852
2022-11-23T02:40:17.3880224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3880439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3880860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3881092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3881497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3881712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3882126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3882339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3882626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3882910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3883355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3883790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3884059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3884325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3885394Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3885962Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:40:17.3887039Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3887277Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:40:17.3887437Z dist init r=0, world=2
2022-11-23T02:40:17.3887569Z dist init r=1, world=2
2022-11-23T02:40:17.3887710Z ok (5.012s)
2022-11-23T02:40:17.3888178Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80935
2022-11-23T02:40:17.3888443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80936
2022-11-23T02:40:17.3888852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3889069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3889492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3889726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3890190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3890415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3890839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3891074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3891358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3891640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3892080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3892517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3892791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3893041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3894112Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3894268Z   world_indices[
2022-11-23T02:40:17.3895341Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3895493Z   world_indices[
2022-11-23T02:40:17.3895645Z dist init r=0, world=2
2022-11-23T02:40:17.3895871Z dist init r=1, world=2
2022-11-23T02:40:17.3896013Z ok (5.014s)
2022-11-23T02:40:17.3896450Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81019
2022-11-23T02:40:17.3896708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81020
2022-11-23T02:40:17.3897102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3897320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3897740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3897975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3898377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3898594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3899012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3899243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3899527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3899793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3900283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3900727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3900997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3901267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3902332Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3902578Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:40:17.3903643Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:40:17.3903889Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:40:17.3904043Z dist init r=1, world=2
2022-11-23T02:40:17.3904196Z dist init r=0, world=2
2022-11-23T02:40:17.3904321Z ok (5.112s)
2022-11-23T02:40:17.3904727Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81103
2022-11-23T02:40:17.3904990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81104
2022-11-23T02:40:17.3905407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3905622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3906038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3906327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3906734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3906930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3907351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3907579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3907870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3908152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3908594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3909038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3909311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3909577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3910285Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3910475Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3911156Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3911341Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3911493Z dist init r=0, world=2
2022-11-23T02:40:17.3911642Z dist init r=1, world=2
2022-11-23T02:40:17.3911783Z ok (4.712s)
2022-11-23T02:40:17.3912039Z   test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded)
2022-11-23T02:40:17.3912304Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.001s)
2022-11-23T02:40:17.3912683Z   test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81187
2022-11-23T02:40:17.3912943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81188
2022-11-23T02:40:17.3913355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3913569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3914022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3914237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3914659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3914889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3915289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3915522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3915806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3916093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3916596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3917028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3917298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3917563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3917718Z dist init r=1, world=2
2022-11-23T02:40:17.3917850Z dist init r=0, world=2
2022-11-23T02:40:17.3917995Z ok (4.813s)
2022-11-23T02:40:17.3918434Z   test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81271
2022-11-23T02:40:17.3918694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81272
2022-11-23T02:40:17.3919110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3919324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3919742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3919972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3920357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3920626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3921046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3921273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3921556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3921842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3922282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3922712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3922983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3923235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3923897Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:40:17.3924057Z   warnings.warn(
2022-11-23T02:40:17.3924716Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:40:17.3924868Z   warnings.warn(
2022-11-23T02:40:17.3925021Z dist init r=1, world=2
2022-11-23T02:40:17.3925170Z dist init r=0, world=2
2022-11-23T02:40:17.3925311Z ok (4.612s)
2022-11-23T02:40:17.3925833Z   test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81355
2022-11-23T02:40:17.3926113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81356
2022-11-23T02:40:17.3926533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3926749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3927253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3927485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3927887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3928103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3928516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3928727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3929014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3929297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3929733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3930178Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3930448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3930713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3930866Z dist init r=1, world=2
2022-11-23T02:40:17.3931016Z dist init r=0, world=2
2022-11-23T02:40:17.3931141Z ok (4.711s)
2022-11-23T02:40:17.3931597Z   test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81439
2022-11-23T02:40:17.3931864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81440
2022-11-23T02:40:17.3932272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3932491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3932913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3933145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3933552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3933748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3934172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3934402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3934686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3934971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3935411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3935848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3936117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3936386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3936524Z dist init r=0, world=2
2022-11-23T02:40:17.3936680Z dist init r=1, world=2
2022-11-23T02:40:17.3936819Z ok (4.711s)
2022-11-23T02:40:17.3937216Z   test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81523
2022-11-23T02:40:17.3937546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81524
2022-11-23T02:40:17.3937961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3938176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3938593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3938807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3939217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3939433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3939844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3940075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3940358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3940639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3941079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3941492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3941814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3942088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3942242Z dist init r=1, world=2
2022-11-23T02:40:17.3942395Z dist init r=0, world=2
2022-11-23T02:40:17.3942540Z ok (4.711s)
2022-11-23T02:40:17.3942924Z   test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81607
2022-11-23T02:40:17.3943181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81608
2022-11-23T02:40:17.3943572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3943792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3944209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3944442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3944844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3945056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3945469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3945696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3945980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3946244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:40:17.3946685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3947123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:40:17.3947391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:40:17.3947654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3947866Z dist init r=0, world=2
2022-11-23T02:40:17.3948015Z dist init r=1, world=2
2022-11-23T02:40:17.3948154Z ok (4.912s)
2022-11-23T02:40:17.3948527Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81691
2022-11-23T02:40:17.3948942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3949160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3949580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3949809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3950090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3950524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:40:17.3950795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3950948Z dist init r=0, world=1
2022-11-23T02:40:17.3951072Z ok (3.909s)
2022-11-23T02:40:17.3951465Z   test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81733
2022-11-23T02:40:17.3951874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3952139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3952564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3952795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3953083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3953519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:40:17.3953764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3954439Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3954629Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3954782Z dist init r=0, world=1
2022-11-23T02:40:17.3954922Z ok (3.909s)
2022-11-23T02:40:17.3955328Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81775
2022-11-23T02:40:17.3955743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:40:17.3955959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:40:17.3956376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:40:17.3956589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:40:17.3956872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:40:17.3957320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:40:17.3957591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:40:17.3958265Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:40:17.3958504Z   warnings.warn(message, UserWarning)
2022-11-23T02:40:17.3958658Z dist init r=0, world=1
2022-11-23T02:40:17.3958798Z ok (4.009s)
2022-11-23T02:40:17.3958822Z 
2022-11-23T02:40:17.3959135Z ----------------------------------------------------------------------
2022-11-23T02:40:17.3959274Z Ran 52 tests in 252.036s
2022-11-23T02:40:17.3959294Z 
2022-11-23T02:40:17.3959441Z OK (skipped=1)
2022-11-23T02:40:17.3959460Z 
2022-11-23T02:40:17.3959624Z Generating XML reports...
2022-11-23T02:40:17.3960181Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221123023604.xml
2022-11-23T02:40:17.3960730Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221123023604.xml
2022-11-23T02:40:17.3960754Z 
2022-11-23T02:40:17.3961279Z ##[endgroup]
2022-11-23T02:40:17.3961827Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_p88flszp)
2022-11-23T02:40:17.3961847Z 
2022-11-23T02:40:17.3962178Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2022-11-23 02:40:17.301814]
2022-11-23T02:40:17.3962744Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:40:17.302164]
2022-11-23T02:58:46.8983828Z 
2022-11-23T02:58:46.8984658Z Expand the folded group to see the log file of distributed/rpc/cuda/test_tensorpipe_agent
2022-11-23T02:58:46.8987131Z ##[group]PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_lf2dvlz7)
2022-11-23T02:58:46.8988756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7vzz6aiu
2022-11-23T02:58:46.8989799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7vzz6aiu/_remote_module_non_scriptable.py
2022-11-23T02:58:46.8990927Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote_gpu>]>
2022-11-23T02:58:46.8991881Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest)
2022-11-23T02:58:46.8993254Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_simple>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]>
2022-11-23T02:58:46.8994649Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest)
2022-11-23T02:58:46.8995411Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest)
2022-11-23T02:58:46.8996193Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest)
2022-11-23T02:58:46.8997674Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]>
2022-11-23T02:58:46.9001073Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-11-23T02:58:46.9001829Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest)
2022-11-23T02:58:46.9002391Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest)
2022-11-23T02:58:46.9002894Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-11-23T02:58:46.9003447Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRpcTest testMethod=test_profiler_remote_cuda>]>
2022-11-23T02:58:46.9004385Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest)
2022-11-23T02:58:46.9007525Z <unittest.suite.TestSuite tests=[<__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]>
2022-11-23T02:58:46.9009293Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:58:46.9009945Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:58:46.9010836Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:58:46.9011921Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:58:46.9012588Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:58:46.9013279Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:58:46.9014286Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:58:46.9015186Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-11-23T02:58:46.9043158Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_nested_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]>
2022-11-23T02:58:46.9066836Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9067889Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9068868Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9069897Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9070876Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9071928Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9072966Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9074074Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9075147Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9077752Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9078681Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9079604Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9080563Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9081498Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9082458Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9083393Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9084262Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9085164Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9086507Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9087196Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9087731Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9088434Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9089017Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9089576Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9090163Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9090918Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9091765Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9092837Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9093796Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9094728Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9095663Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9096622Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9097508Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9098421Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9099328Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9100302Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9101357Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9102256Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9103142Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9104090Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9105012Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9105902Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9106889Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9107897Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9108876Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9109778Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9110671Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9111649Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9112616Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9113617Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9114550Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9115538Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9116525Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9117601Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9118647Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9119672Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9120643Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9121674Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9122759Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9123659Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9124583Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9125487Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9126832Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9127718Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9128659Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9129532Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9130404Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9131299Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9132301Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9133310Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9134270Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9135256Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9136197Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9137298Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9138218Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9139150Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9140290Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9141047Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9141631Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9142211Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9142786Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9143342Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9143919Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9144498Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9145054Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9145632Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9146212Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T02:58:46.9147200Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_device_maps_backward_pass>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]>
2022-11-23T02:58:46.9148184Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-11-23T02:58:46.9148767Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-11-23T02:58:46.9149371Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-11-23T02:58:46.9150232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9150852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9151516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9152064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9152610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphrb1p4kb
2022-11-23T02:58:46.9153214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphrb1p4kb/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9153560Z 
2022-11-23T02:58:46.9153707Z Running tests...
2022-11-23T02:58:46.9154198Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9154835Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9155505Z   test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9156122Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81885
2022-11-23T02:58:46.9156644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81886
2022-11-23T02:58:46.9157147Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81887
2022-11-23T02:58:46.9157658Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81888
2022-11-23T02:58:46.9158349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9158933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9159580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9160127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9160781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9161296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9161943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9162485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9163138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9163638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9164287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9164825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9165460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9166302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9166954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9167492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9168017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9q1cmes4
2022-11-23T02:58:46.9168631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9q1cmes4/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9169245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqiduh_4
2022-11-23T02:58:46.9169858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqiduh_4/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9170450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfs0gokul
2022-11-23T02:58:46.9171167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfs0gokul/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9171768Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsx9r9q0
2022-11-23T02:58:46.9172359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsx9r9q0/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9172941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9173487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9174034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9174563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9175024Z skip: Need at least 4 CUDA devices (4.439s)
2022-11-23T02:58:46.9175257Z 
2022-11-23T02:58:46.9175573Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9175965Z Ran 1 test in 4.439s
2022-11-23T02:58:46.9176165Z 
2022-11-23T02:58:46.9176312Z OK (skipped=1)
2022-11-23T02:58:46.9176503Z 
2022-11-23T02:58:46.9176664Z Generating XML reports...
2022-11-23T02:58:46.9177421Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20221123024021.xml
2022-11-23T02:58:46.9178255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9178785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9179502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9180066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9180590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlqrtgg4
2022-11-23T02:58:46.9181213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlqrtgg4/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9181553Z 
2022-11-23T02:58:46.9181701Z Running tests...
2022-11-23T02:58:46.9182168Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9182816Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9183436Z   test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9184007Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82056
2022-11-23T02:58:46.9184514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82057
2022-11-23T02:58:46.9185025Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82058
2022-11-23T02:58:46.9185546Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82059
2022-11-23T02:58:46.9186219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9186746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9187394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9187936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9188575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9189105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9189752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9190278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9190928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9191508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9192155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9192677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9193329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9193846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9194498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9195021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9195558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjf7omp1t
2022-11-23T02:58:46.9196181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjf7omp1t/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9196779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpffsaf5mp
2022-11-23T02:58:46.9197395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpffsaf5mp/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9198006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_r96uahx
2022-11-23T02:58:46.9198608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_r96uahx/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9199250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph1u2y7p1
2022-11-23T02:58:46.9199866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph1u2y7p1/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9200446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9200998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9201522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9202054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9202528Z fi_getinfo: -61
2022-11-23T02:58:46.9202874Z fi_getinfo: -61
2022-11-23T02:58:46.9203216Z fi_getinfo: -61
2022-11-23T02:58:46.9203570Z fi_getinfo: -61
2022-11-23T02:58:46.9203870Z ok (5.838s)
2022-11-23T02:58:46.9204055Z 
2022-11-23T02:58:46.9204358Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9204763Z Ran 1 test in 5.839s
2022-11-23T02:58:46.9204961Z 
2022-11-23T02:58:46.9205074Z OK
2022-11-23T02:58:46.9205246Z 
2022-11-23T02:58:46.9205405Z Generating XML reports...
2022-11-23T02:58:46.9206506Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123024028.xml
2022-11-23T02:58:46.9207359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9207872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9208521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9209065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9209590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9x0cdss
2022-11-23T02:58:46.9210209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9x0cdss/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9210551Z 
2022-11-23T02:58:46.9210699Z Running tests...
2022-11-23T02:58:46.9211184Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9211816Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9212552Z   test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9213137Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82479
2022-11-23T02:58:46.9213661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82480
2022-11-23T02:58:46.9214161Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82481
2022-11-23T02:58:46.9214667Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82482
2022-11-23T02:58:46.9215360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9215873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9216525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9217068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9217724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9218227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9218873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9219410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9220112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9220643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9221291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9221826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9222460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9222980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9223625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9224162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9224685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58rui7l1
2022-11-23T02:58:46.9225299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58rui7l1/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9225905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5q7pzti
2022-11-23T02:58:46.9226502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5q7pzti/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9227112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7pcv_w7
2022-11-23T02:58:46.9227716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7pcv_w7/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9228320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_1ve5de
2022-11-23T02:58:46.9228902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_1ve5de/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9229477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9230030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9230553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9231094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9231617Z fi_getinfo: -61
2022-11-23T02:58:46.9231967Z fi_getinfo: -61
2022-11-23T02:58:46.9232303Z fi_getinfo: -61
2022-11-23T02:58:46.9232693Z fi_getinfo: -61
2022-11-23T02:58:46.9233005Z ok (5.922s)
2022-11-23T02:58:46.9233171Z 
2022-11-23T02:58:46.9233478Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9233880Z Ran 1 test in 5.923s
2022-11-23T02:58:46.9234078Z 
2022-11-23T02:58:46.9234207Z OK
2022-11-23T02:58:46.9234377Z 
2022-11-23T02:58:46.9234523Z Generating XML reports...
2022-11-23T02:58:46.9235279Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123024037.xml
2022-11-23T02:58:46.9236121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9236649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9237285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9237839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9238380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoyq2_pjh
2022-11-23T02:58:46.9238979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoyq2_pjh/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9239318Z 
2022-11-23T02:58:46.9239466Z Running tests...
2022-11-23T02:58:46.9239948Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9240646Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9241291Z   test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9241894Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82902
2022-11-23T02:58:46.9242421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82903
2022-11-23T02:58:46.9242921Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82904
2022-11-23T02:58:46.9243437Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82905
2022-11-23T02:58:46.9244118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9244646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9245284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9246151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9246820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9247340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9247968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9248489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9249133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9249665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9250322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9250866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9251516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9252019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9252665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9253300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9253817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhqwedhz
2022-11-23T02:58:46.9254442Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhqwedhz/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9255051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp55rk6p1
2022-11-23T02:58:46.9255664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp55rk6p1/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9256253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fjgt1n1
2022-11-23T02:58:46.9256857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fjgt1n1/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9257458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppb2g5gu9
2022-11-23T02:58:46.9258067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppb2g5gu9/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9258636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9259186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9259725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9260251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9260788Z fi_getinfo: -61
2022-11-23T02:58:46.9261152Z fi_getinfo: -61
2022-11-23T02:58:46.9261484Z fi_getinfo: -61
2022-11-23T02:58:46.9261833Z fi_getinfo: -61
2022-11-23T02:58:46.9262142Z ok (6.002s)
2022-11-23T02:58:46.9262326Z 
2022-11-23T02:58:46.9262629Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9263022Z Ran 1 test in 6.002s
2022-11-23T02:58:46.9263219Z 
2022-11-23T02:58:46.9263349Z OK
2022-11-23T02:58:46.9263522Z 
2022-11-23T02:58:46.9263687Z Generating XML reports...
2022-11-23T02:58:46.9264425Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123024046.xml
2022-11-23T02:58:46.9265270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9265793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9266446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9266978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9267520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjyrtfg7
2022-11-23T02:58:46.9268130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjyrtfg7/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9268477Z 
2022-11-23T02:58:46.9268607Z Running tests...
2022-11-23T02:58:46.9269085Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9269731Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9270373Z   test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9270948Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83325
2022-11-23T02:58:46.9271478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83326
2022-11-23T02:58:46.9272160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9272667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9273381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9273923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9274578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9275083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9275730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9276275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9276820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcm112961
2022-11-23T02:58:46.9277414Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcm112961/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9278016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkug5_vs3
2022-11-23T02:58:46.9278624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkug5_vs3/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9279185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9279734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9280204Z fi_getinfo: -61
2022-11-23T02:58:46.9280541Z fi_getinfo: -61
2022-11-23T02:58:46.9280854Z ok (5.304s)
2022-11-23T02:58:46.9281038Z 
2022-11-23T02:58:46.9281389Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9281799Z Ran 1 test in 5.304s
2022-11-23T02:58:46.9281981Z 
2022-11-23T02:58:46.9282112Z OK
2022-11-23T02:58:46.9282284Z 
2022-11-23T02:58:46.9282443Z Generating XML reports...
2022-11-23T02:58:46.9283193Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024054.xml
2022-11-23T02:58:46.9284026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9284551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9285200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9286072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9286606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6e0nw9q6
2022-11-23T02:58:46.9287221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6e0nw9q6/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9287560Z 
2022-11-23T02:58:46.9287708Z Running tests...
2022-11-23T02:58:46.9288175Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9288827Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9289485Z   test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9290086Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83515
2022-11-23T02:58:46.9290595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83516
2022-11-23T02:58:46.9291275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9291807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9292463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9292994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9293646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9294254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9294888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9295431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9295971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgl8vs1c8
2022-11-23T02:58:46.9296586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgl8vs1c8/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9297181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb649pyr7
2022-11-23T02:58:46.9297795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb649pyr7/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9298373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9298908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9299435Z fi_getinfo: -61
2022-11-23T02:58:46.9299787Z fi_getinfo: -61
2022-11-23T02:58:46.9300368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb649pyr7/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:58:46.9301154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgl8vs1c8/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:58:46.9302030Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpgl8vs1c8/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T02:58:46.9302600Z ok (5.531s)
2022-11-23T02:58:46.9302786Z 
2022-11-23T02:58:46.9303095Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9303487Z Ran 1 test in 5.532s
2022-11-23T02:58:46.9303685Z 
2022-11-23T02:58:46.9303816Z OK
2022-11-23T02:58:46.9303989Z 
2022-11-23T02:58:46.9304151Z Generating XML reports...
2022-11-23T02:58:46.9304885Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024102.xml
2022-11-23T02:58:46.9305732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9306258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9306910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9307444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9307987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_fx7ovhu
2022-11-23T02:58:46.9308597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_fx7ovhu/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9308937Z 
2022-11-23T02:58:46.9309086Z Running tests...
2022-11-23T02:58:46.9309550Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9310199Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9310827Z   test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9311390Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83721
2022-11-23T02:58:46.9311916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83722
2022-11-23T02:58:46.9312601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9313125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9313757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9314379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9315038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9315537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9316185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9316724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9317270Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_orgyasr
2022-11-23T02:58:46.9317865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_orgyasr/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9318474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp867ir7y9
2022-11-23T02:58:46.9319081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp867ir7y9/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9319648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9320194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9320657Z fi_getinfo: -61
2022-11-23T02:58:46.9321013Z fi_getinfo: -61
2022-11-23T02:58:46.9321346Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:58:46.9339693Z RuntimeError('CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fcf78aa259b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fcf78a9ddfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7fcf78d2806e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #3: <unknown function> + 0x17c9d (0x7fcf78d00c9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #4: <unknown function> + 0xdf449d (0x7fcf79d4149d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #5: <unknown function> + 0x29c8775 (0x7fcf7b915775 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #6: <unknown function> + 0x29c891b (0x7fcf7b91591b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7fcf85ad9cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: <unknown function> + 0x20c43b5 (0x7fcf85dfb3b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7fcf85b14e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x12691af (0x7fcf84fa01af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7fcf85336641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x22a6e23 (0x7fcf85fdde23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7fcf857fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x20c8908 (0x7fcf85dff908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7fcf857fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: <unknown function> + 0x3443e11 (0x7fcf8717ae11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x34443bb (0x7fcf8717b3bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7fcf85853de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7fcf8532e05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #20: <unknown function> + 0x2471e09 (0x7fcf861a8e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7fcf859c53f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #22: <unknown function> + 0x36467f (0x7fcf90a7a67f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #23: <unknown function> + 0x364b3c (0x7fcf90a7ab3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #24: <unknown function> + 0x1ddc68 (0x564fe9d54c68 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0x1049f3 (0x564fe9c7b9f3 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0x104425 (0x564fe9c7b425 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x18fc9b (0x564fe9d06c9b in /opt/conda/bin/python)\nframe #30: <unknown function> + 0x1052a5 (0x564fe9c7c2a5 in /opt/conda/bin/python)\nframe #31: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #32: <unknown function> + 0x18fc9b (0x564fe9d06c9b in /opt/conda/bin/python)\nframe #33: <unknown function> + 0x1052a5 (0x564fe9c7c2a5 in /opt/conda/bin/python)\nframe #34: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x20a (0x564fe9cbefaa in /opt/conda/bin/python)\nframe #36: _PyEval_EvalFrameDefault + 0x26e4 (0x564fe9d5a774 in /opt/conda/bin/python)\nframe #37: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #38: _PyObject_Call + 0x20a (0x564fe9cbefaa in /opt/conda/bin/python)\nframe #39: <unknown function> + 0xaa8dba (0x7fcf911bedba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fcf911bcffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fcf911c02d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7fcf911c09a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7fcf887625d4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fcf911c00c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #45: <unknown function> + 0x4a249d3 (0x7fcf8875b9d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fcf8875c568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fcf88756867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #48: <unknown function> + 0x4a54552 (0x7fcf8878b552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fcf78a9090b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #50: <unknown function> + 0xdbbf4 (0x7fcfa86a0bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #51: <unknown function> + 0x76db (0x7fcfc8cfb6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #52: clone + 0x3f (0x7fcfc8a2461f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T02:58:46.9350447Z Traceback (most recent call last):
2022-11-23T02:58:46.9351076Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:58:46.9351594Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:58:46.9352251Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 92, in _create_module
2022-11-23T02:58:46.9352725Z     module.to(device)
2022-11-23T02:58:46.9353248Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1120, in to
2022-11-23T02:58:46.9353698Z     return self._apply(convert)
2022-11-23T02:58:46.9354255Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 795, in _apply
2022-11-23T02:58:46.9354709Z     param_applied = fn(param)
2022-11-23T02:58:46.9355252Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1118, in convert
2022-11-23T02:58:46.9355797Z     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2022-11-23T02:58:46.9356273Z RuntimeError: CUDA error: invalid device ordinal
2022-11-23T02:58:46.9356791Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-11-23T02:58:46.9357326Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-11-23T02:58:46.9357898Z Exception raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):
2022-11-23T02:58:46.9358919Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fcf78aa259b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:46.9359965Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fcf78a9ddfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:46.9360947Z frame #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7fcf78d2806e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T02:58:46.9361734Z frame #3: <unknown function> + 0x17c9d (0x7fcf78d00c9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T02:58:46.9362456Z frame #4: <unknown function> + 0xdf449d (0x7fcf79d4149d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:46.9363167Z frame #5: <unknown function> + 0x29c8775 (0x7fcf7b915775 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:46.9363883Z frame #6: <unknown function> + 0x29c891b (0x7fcf7b91591b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:46.9365026Z frame #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7fcf85ad9cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9366298Z frame #8: <unknown function> + 0x20c43b5 (0x7fcf85dfb3b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9367340Z frame #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7fcf85b14e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9368238Z frame #10: <unknown function> + 0x12691af (0x7fcf84fa01af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9369250Z frame #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7fcf85336641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9370139Z frame #12: <unknown function> + 0x22a6e23 (0x7fcf85fdde23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9371241Z frame #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7fcf857fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9372177Z frame #14: <unknown function> + 0x20c8908 (0x7fcf85dff908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9373268Z frame #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7fcf857fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9374184Z frame #16: <unknown function> + 0x3443e11 (0x7fcf8717ae11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9374901Z frame #17: <unknown function> + 0x34443bb (0x7fcf8717b3bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9376022Z frame #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7fcf85853de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9377216Z frame #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7fcf8532e05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9378102Z frame #20: <unknown function> + 0x2471e09 (0x7fcf861a8e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9379147Z frame #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7fcf859c53f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9380067Z frame #22: <unknown function> + 0x36467f (0x7fcf90a7a67f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9380797Z frame #23: <unknown function> + 0x364b3c (0x7fcf90a7ab3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9381343Z frame #24: <unknown function> + 0x1ddc68 (0x564fe9d54c68 in /opt/conda/bin/python)
2022-11-23T02:58:46.9381810Z frame #25: <unknown function> + 0x1049f3 (0x564fe9c7b9f3 in /opt/conda/bin/python)
2022-11-23T02:58:46.9382354Z frame #26: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9382846Z frame #27: <unknown function> + 0x104425 (0x564fe9c7b425 in /opt/conda/bin/python)
2022-11-23T02:58:46.9383317Z frame #28: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9383773Z frame #29: <unknown function> + 0x18fc9b (0x564fe9d06c9b in /opt/conda/bin/python)
2022-11-23T02:58:46.9384256Z frame #30: <unknown function> + 0x1052a5 (0x564fe9c7c2a5 in /opt/conda/bin/python)
2022-11-23T02:58:46.9384729Z frame #31: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9385178Z frame #32: <unknown function> + 0x18fc9b (0x564fe9d06c9b in /opt/conda/bin/python)
2022-11-23T02:58:46.9385651Z frame #33: <unknown function> + 0x1052a5 (0x564fe9c7c2a5 in /opt/conda/bin/python)
2022-11-23T02:58:46.9386116Z frame #34: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9386590Z frame #35: _PyObject_Call + 0x20a (0x564fe9cbefaa in /opt/conda/bin/python)
2022-11-23T02:58:46.9387071Z frame #36: _PyEval_EvalFrameDefault + 0x26e4 (0x564fe9d5a774 in /opt/conda/bin/python)
2022-11-23T02:58:46.9387563Z frame #37: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9388034Z frame #38: _PyObject_Call + 0x20a (0x564fe9cbefaa in /opt/conda/bin/python)
2022-11-23T02:58:46.9388728Z frame #39: <unknown function> + 0xaa8dba (0x7fcf911bedba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9389593Z frame #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fcf911bcffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9390694Z frame #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fcf911c02d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9391903Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7fcf911c09a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9393271Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7fcf887625d4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9394629Z frame #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fcf911c00c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9395598Z frame #45: <unknown function> + 0x4a249d3 (0x7fcf8875b9d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9396589Z frame #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fcf8875c568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9397740Z frame #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fcf88756867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9398604Z frame #48: <unknown function> + 0x4a54552 (0x7fcf8878b552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9399450Z frame #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fcf78a9090b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:46.9400035Z frame #50: <unknown function> + 0xdbbf4 (0x7fcfa86a0bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:58:46.9400676Z frame #51: <unknown function> + 0x76db (0x7fcfc8cfb6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:58:46.9401262Z frame #52: clone + 0x3f (0x7fcfc8a2461f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:58:46.9401540Z 
2022-11-23T02:58:46.9401559Z 
2022-11-23T02:58:46.9401734Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:58:46.9439587Z RuntimeError('On WorkerInfo(id=1, name=worker1):\nRuntimeError(\'CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fcf78aa259b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fcf78a9ddfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7fcf78d2806e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #3: <unknown function> + 0x17c9d (0x7fcf78d00c9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #4: <unknown function> + 0xdf449d (0x7fcf79d4149d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #5: <unknown function> + 0x29c8775 (0x7fcf7b915775 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #6: <unknown function> + 0x29c891b (0x7fcf7b91591b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7fcf85ad9cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: <unknown function> + 0x20c43b5 (0x7fcf85dfb3b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7fcf85b14e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x12691af (0x7fcf84fa01af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7fcf85336641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x22a6e23 (0x7fcf85fdde23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7fcf857fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x20c8908 (0x7fcf85dff908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7fcf857fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: <unknown function> + 0x3443e11 (0x7fcf8717ae11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x34443bb (0x7fcf8717b3bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7fcf85853de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7fcf8532e05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #20: <unknown function> + 0x2471e09 (0x7fcf861a8e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7fcf859c53f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #22: <unknown function> + 0x36467f (0x7fcf90a7a67f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #23: <unknown function> + 0x364b3c (0x7fcf90a7ab3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #24: <unknown function> + 0x1ddc68 (0x564fe9d54c68 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0x1049f3 (0x564fe9c7b9f3 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0x104425 (0x564fe9c7b425 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x18fc9b (0x564fe9d06c9b in /opt/conda/bin/python)\nframe #30: <unknown function> + 0x1052a5 (0x564fe9c7c2a5 in /opt/conda/bin/python)\nframe #31: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #32: <unknown function> + 0x18fc9b (0x564fe9d06c9b in /opt/conda/bin/python)\nframe #33: <unknown function> + 0x1052a5 (0x564fe9c7c2a5 in /opt/conda/bin/python)\nframe #34: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x20a (0x564fe9cbefaa in /opt/conda/bin/python)\nframe #36: _PyEval_EvalFrameDefault + 0x26e4 (0x564fe9d5a774 in /opt/conda/bin/python)\nframe #37: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #38: _PyObject_Call + 0x20a (0x564fe9cbefaa in /opt/conda/bin/python)\nframe #39: <unknown function> + 0xaa8dba (0x7fcf911bedba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fcf911bcffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fcf911c02d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7fcf911c09a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7fcf887625d4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fcf911c00c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #45: <unknown function> + 0x4a249d3 (0x7fcf8875b9d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fcf8875c568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fcf88756867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #48: <unknown function> + 0x4a54552 (0x7fcf8878b552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fcf78a9090b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #50: <unknown function> + 0xdbbf4 (0x7fcfa86a0bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #51: <unknown function> + 0x76db (0x7fcfc8cfb6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #52: clone + 0x3f (0x7fcfc8a2461f in /lib/x86_64-linux-gnu/libc.so.6)\n\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 92, in _create_module\n    module.to(device)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1120, in to\n    return self._apply(convert)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 795, in _apply\n    param_applied = fn(param)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1118, in convert\n    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)\nRuntimeError: CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fcf78aa259b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fcf78a9ddfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7fcf78d2806e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #3: <unknown function> + 0x17c9d (0x7fcf78d00c9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #4: <unknown function> + 0xdf449d (0x7fcf79d4149d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #5: <unknown function> + 0x29c8775 (0x7fcf7b915775 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #6: <unknown function> + 0x29c891b (0x7fcf7b91591b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7fcf85ad9cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: <unknown function> + 0x20c43b5 (0x7fcf85dfb3b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7fcf85b14e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x12691af (0x7fcf84fa01af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7fcf85336641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x22a6e23 (0x7fcf85fdde23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7fcf857fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x20c8908 (0x7fcf85dff908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7fcf857fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: <unknown function> + 0x3443e11 (0x7fcf8717ae11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x34443bb (0x7fcf8717b3bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7fcf85853de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7fcf8532e05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #20: <unknown function> + 0x2471e09 (0x7fcf861a8e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7fcf859c53f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #22: <unknown function> + 0x36467f (0x7fcf90a7a67f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #23: <unknown function> + 0x364b3c (0x7fcf90a7ab3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #24: <unknown function> + 0x1ddc68 (0x564fe9d54c68 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0x1049f3 (0x564fe9c7b9f3 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0x104425 (0x564fe9c7b425 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x18fc9b (0x564fe9d06c9b in /opt/conda/bin/python)\nframe #30: <unknown function> + 0x1052a5 (0x564fe9c7c2a5 in /opt/conda/bin/python)\nframe #31: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #32: <unknown function> + 0x18fc9b (0x564fe9d06c9b in /opt/conda/bin/python)\nframe #33: <unknown function> + 0x1052a5 (0x564fe9c7c2a5 in /opt/conda/bin/python)\nframe #34: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x20a (0x564fe9cbefaa in /opt/conda/bin/python)\nframe #36: _PyEval_EvalFrameDefault + 0x26e4 (0x564fe9d5a774 in /opt/conda/bin/python)\nframe #37: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)\nframe #38: _PyObject_Call + 0x20a (0x564fe9cbefaa in /opt/conda/bin/python)\nframe #39: <unknown function> + 0xaa8dba (0x7fcf911bedba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fcf911bcffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fcf911c02d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7fcf911c09a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7fcf887625d4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fcf911c00c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #45: <unknown function> + 0x4a249d3 (0x7fcf8875b9d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fcf8875c568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fcf88756867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #48: <unknown function> + 0x4a54552 (0x7fcf8878b552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fcf78a9090b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #50: <unknown function> + 0xdbbf4 (0x7fcfa86a0bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #51: <unknown function> + 0x76db (0x7fcfc8cfb6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #52: clone + 0x3f (0x7fcfc8a2461f in /lib/x86_64-linux-gnu/libc.so.6)\n\n')
2022-11-23T02:58:46.9462340Z Traceback (most recent call last):
2022-11-23T02:58:46.9462967Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:58:46.9463510Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:58:46.9463998Z   File "/tmp/tmp_fx7ovhu/_remote_module_non_scriptable.py", line 47, in _remote_forward
2022-11-23T02:58:46.9464440Z     module = module_rref.local_value()
2022-11-23T02:58:46.9465050Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T02:58:46.9465505Z     raise exc
2022-11-23T02:58:46.9465855Z RuntimeError: On WorkerInfo(id=1, name=worker1):
2022-11-23T02:58:46.9466403Z RuntimeError('CUDA error: invalid device ordinal
2022-11-23T02:58:46.9466950Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-11-23T02:58:46.9467469Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-11-23T02:58:46.9468046Z Exception raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):
2022-11-23T02:58:46.9468995Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fcf78aa259b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:46.9470064Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fcf78a9ddfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:46.9471045Z frame #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7fcf78d2806e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T02:58:46.9471807Z frame #3: <unknown function> + 0x17c9d (0x7fcf78d00c9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T02:58:46.9472525Z frame #4: <unknown function> + 0xdf449d (0x7fcf79d4149d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:46.9473254Z frame #5: <unknown function> + 0x29c8775 (0x7fcf7b915775 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:46.9473964Z frame #6: <unknown function> + 0x29c891b (0x7fcf7b91591b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:46.9475053Z frame #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7fcf85ad9cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9475988Z frame #8: <unknown function> + 0x20c43b5 (0x7fcf85dfb3b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9477018Z frame #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7fcf85b14e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9477991Z frame #10: <unknown function> + 0x12691af (0x7fcf84fa01af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9479027Z frame #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7fcf85336641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9479897Z frame #12: <unknown function> + 0x22a6e23 (0x7fcf85fdde23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9480996Z frame #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7fcf857fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9481925Z frame #14: <unknown function> + 0x20c8908 (0x7fcf85dff908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9483062Z frame #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7fcf857fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9483999Z frame #16: <unknown function> + 0x3443e11 (0x7fcf8717ae11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9484693Z frame #17: <unknown function> + 0x34443bb (0x7fcf8717b3bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9485972Z frame #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7fcf85853de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9487349Z frame #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7fcf8532e05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9488248Z frame #20: <unknown function> + 0x2471e09 (0x7fcf861a8e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9489307Z frame #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7fcf859c53f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9490232Z frame #22: <unknown function> + 0x36467f (0x7fcf90a7a67f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9490949Z frame #23: <unknown function> + 0x364b3c (0x7fcf90a7ab3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9491497Z frame #24: <unknown function> + 0x1ddc68 (0x564fe9d54c68 in /opt/conda/bin/python)
2022-11-23T02:58:46.9491979Z frame #25: <unknown function> + 0x1049f3 (0x564fe9c7b9f3 in /opt/conda/bin/python)
2022-11-23T02:58:46.9492439Z frame #26: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9492913Z frame #27: <unknown function> + 0x104425 (0x564fe9c7b425 in /opt/conda/bin/python)
2022-11-23T02:58:46.9493385Z frame #28: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9493968Z frame #29: <unknown function> + 0x18fc9b (0x564fe9d06c9b in /opt/conda/bin/python)
2022-11-23T02:58:46.9494429Z frame #30: <unknown function> + 0x1052a5 (0x564fe9c7c2a5 in /opt/conda/bin/python)
2022-11-23T02:58:46.9494900Z frame #31: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9495369Z frame #32: <unknown function> + 0x18fc9b (0x564fe9d06c9b in /opt/conda/bin/python)
2022-11-23T02:58:46.9495817Z frame #33: <unknown function> + 0x1052a5 (0x564fe9c7c2a5 in /opt/conda/bin/python)
2022-11-23T02:58:46.9496280Z frame #34: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9496752Z frame #35: _PyObject_Call + 0x20a (0x564fe9cbefaa in /opt/conda/bin/python)
2022-11-23T02:58:46.9497253Z frame #36: _PyEval_EvalFrameDefault + 0x26e4 (0x564fe9d5a774 in /opt/conda/bin/python)
2022-11-23T02:58:46.9497728Z frame #37: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9498201Z frame #38: _PyObject_Call + 0x20a (0x564fe9cbefaa in /opt/conda/bin/python)
2022-11-23T02:58:46.9498891Z frame #39: <unknown function> + 0xaa8dba (0x7fcf911bedba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9499773Z frame #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fcf911bcffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9500910Z frame #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fcf911c02d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9502152Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7fcf911c09a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9503468Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7fcf887625d4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9504827Z frame #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fcf911c00c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9505799Z frame #45: <unknown function> + 0x4a249d3 (0x7fcf8875b9d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9506817Z frame #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fcf8875c568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9507948Z frame #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fcf88756867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9508818Z frame #48: <unknown function> + 0x4a54552 (0x7fcf8878b552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9509587Z frame #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fcf78a9090b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:46.9510178Z frame #50: <unknown function> + 0xdbbf4 (0x7fcfa86a0bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:58:46.9510795Z frame #51: <unknown function> + 0x76db (0x7fcfc8cfb6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:58:46.9511445Z frame #52: clone + 0x3f (0x7fcfc8a2461f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:58:46.9511851Z ')
2022-11-23T02:58:46.9512183Z Traceback (most recent call last):
2022-11-23T02:58:46.9512762Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:58:46.9513293Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:58:46.9513946Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 92, in _create_module
2022-11-23T02:58:46.9514395Z     module.to(device)
2022-11-23T02:58:46.9514928Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1120, in to
2022-11-23T02:58:46.9515374Z     return self._apply(convert)
2022-11-23T02:58:46.9515932Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 795, in _apply
2022-11-23T02:58:46.9516363Z     param_applied = fn(param)
2022-11-23T02:58:46.9516925Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1118, in convert
2022-11-23T02:58:46.9517472Z     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2022-11-23T02:58:46.9517932Z RuntimeError: CUDA error: invalid device ordinal
2022-11-23T02:58:46.9518462Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-11-23T02:58:46.9518994Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-11-23T02:58:46.9519617Z Exception raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):
2022-11-23T02:58:46.9520556Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fcf78aa259b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:46.9521615Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fcf78a9ddfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:46.9522588Z frame #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7fcf78d2806e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T02:58:46.9523363Z frame #3: <unknown function> + 0x17c9d (0x7fcf78d00c9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T02:58:46.9524073Z frame #4: <unknown function> + 0xdf449d (0x7fcf79d4149d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:46.9524807Z frame #5: <unknown function> + 0x29c8775 (0x7fcf7b915775 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:46.9525520Z frame #6: <unknown function> + 0x29c891b (0x7fcf7b91591b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:46.9526902Z frame #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7fcf85ad9cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9527839Z frame #8: <unknown function> + 0x20c43b5 (0x7fcf85dfb3b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9528861Z frame #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7fcf85b14e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9529753Z frame #10: <unknown function> + 0x12691af (0x7fcf84fa01af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9530890Z frame #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7fcf85336641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9531782Z frame #12: <unknown function> + 0x22a6e23 (0x7fcf85fdde23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9532936Z frame #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7fcf857fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9533858Z frame #14: <unknown function> + 0x20c8908 (0x7fcf85dff908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9534954Z frame #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7fcf857fa0e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9535878Z frame #16: <unknown function> + 0x3443e11 (0x7fcf8717ae11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9536597Z frame #17: <unknown function> + 0x34443bb (0x7fcf8717b3bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9537693Z frame #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7fcf85853de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9538903Z frame #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7fcf8532e05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9539777Z frame #20: <unknown function> + 0x2471e09 (0x7fcf861a8e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9540837Z frame #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7fcf859c53f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9541750Z frame #22: <unknown function> + 0x36467f (0x7fcf90a7a67f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9542487Z frame #23: <unknown function> + 0x364b3c (0x7fcf90a7ab3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9543023Z frame #24: <unknown function> + 0x1ddc68 (0x564fe9d54c68 in /opt/conda/bin/python)
2022-11-23T02:58:46.9543507Z frame #25: <unknown function> + 0x1049f3 (0x564fe9c7b9f3 in /opt/conda/bin/python)
2022-11-23T02:58:46.9543986Z frame #26: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9544461Z frame #27: <unknown function> + 0x104425 (0x564fe9c7b425 in /opt/conda/bin/python)
2022-11-23T02:58:46.9544913Z frame #28: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9545384Z frame #29: <unknown function> + 0x18fc9b (0x564fe9d06c9b in /opt/conda/bin/python)
2022-11-23T02:58:46.9545856Z frame #30: <unknown function> + 0x1052a5 (0x564fe9c7c2a5 in /opt/conda/bin/python)
2022-11-23T02:58:46.9546309Z frame #31: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9546778Z frame #32: <unknown function> + 0x18fc9b (0x564fe9d06c9b in /opt/conda/bin/python)
2022-11-23T02:58:46.9547308Z frame #33: <unknown function> + 0x1052a5 (0x564fe9c7c2a5 in /opt/conda/bin/python)
2022-11-23T02:58:46.9547778Z frame #34: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9548229Z frame #35: _PyObject_Call + 0x20a (0x564fe9cbefaa in /opt/conda/bin/python)
2022-11-23T02:58:46.9548724Z frame #36: _PyEval_EvalFrameDefault + 0x26e4 (0x564fe9d5a774 in /opt/conda/bin/python)
2022-11-23T02:58:46.9549211Z frame #37: <unknown function> + 0x18f742 (0x564fe9d06742 in /opt/conda/bin/python)
2022-11-23T02:58:46.9549664Z frame #38: _PyObject_Call + 0x20a (0x564fe9cbefaa in /opt/conda/bin/python)
2022-11-23T02:58:46.9550356Z frame #39: <unknown function> + 0xaa8dba (0x7fcf911bedba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9551232Z frame #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fcf911bcffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9552331Z frame #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fcf911c02d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9553549Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7fcf911c09a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9554904Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7fcf887625d4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9556255Z frame #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fcf911c00c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:46.9557235Z frame #45: <unknown function> + 0x4a249d3 (0x7fcf8875b9d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9558246Z frame #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fcf8875c568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9559382Z frame #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fcf88756867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9560256Z frame #48: <unknown function> + 0x4a54552 (0x7fcf8878b552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:46.9561003Z frame #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fcf78a9090b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:46.9561595Z frame #50: <unknown function> + 0xdbbf4 (0x7fcfa86a0bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:58:46.9562236Z frame #51: <unknown function> + 0x76db (0x7fcfc8cfb6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:58:46.9562833Z frame #52: clone + 0x3f (0x7fcfc8a2461f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:58:46.9563103Z 
2022-11-23T02:58:46.9563123Z 
2022-11-23T02:58:46.9563141Z 
2022-11-23T02:58:46.9563265Z ok (4.656s)
2022-11-23T02:58:46.9563454Z 
2022-11-23T02:58:46.9563766Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9564174Z Ran 1 test in 4.656s
2022-11-23T02:58:46.9564432Z 
2022-11-23T02:58:46.9564564Z OK
2022-11-23T02:58:46.9564717Z 
2022-11-23T02:58:46.9564884Z Generating XML reports...
2022-11-23T02:58:46.9565643Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024110.xml
2022-11-23T02:58:46.9566733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9567246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9567910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9568461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9569005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgfbykwl
2022-11-23T02:58:46.9569608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgfbykwl/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9569957Z 
2022-11-23T02:58:46.9570108Z Running tests...
2022-11-23T02:58:46.9570595Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9571233Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9571858Z   test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9572432Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83910
2022-11-23T02:58:46.9573034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83911
2022-11-23T02:58:46.9573724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9574257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9574912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9575472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9576115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9576641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9577297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9577821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9578369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkdia6l1
2022-11-23T02:58:46.9578993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkdia6l1/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9579602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqz04d6as
2022-11-23T02:58:46.9580201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqz04d6as/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9580794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9581343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9581795Z fi_getinfo: -61
2022-11-23T02:58:46.9582159Z fi_getinfo: -61
2022-11-23T02:58:46.9582468Z ok (5.438s)
2022-11-23T02:58:46.9582659Z 
2022-11-23T02:58:46.9582966Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9583358Z Ran 1 test in 5.438s
2022-11-23T02:58:46.9583558Z 
2022-11-23T02:58:46.9583689Z OK
2022-11-23T02:58:46.9583861Z 
2022-11-23T02:58:46.9584023Z Generating XML reports...
2022-11-23T02:58:46.9584755Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024118.xml
2022-11-23T02:58:46.9585680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9586211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9586871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9587407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9587947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpst_jkjbq
2022-11-23T02:58:46.9588567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpst_jkjbq/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9588909Z 
2022-11-23T02:58:46.9589042Z Running tests...
2022-11-23T02:58:46.9589525Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9590174Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9590799Z   test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9591348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84100
2022-11-23T02:58:46.9591874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84101
2022-11-23T02:58:46.9592396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84102
2022-11-23T02:58:46.9592907Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84103
2022-11-23T02:58:46.9593644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9594176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9594809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9595349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9596005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9596523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9597161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9597697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9598348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9598848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9599549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9600080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9600731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9601230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9601875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9602410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9602930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmc3hq9fg
2022-11-23T02:58:46.9603550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmc3hq9fg/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9604156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5nurj2sd
2022-11-23T02:58:46.9604766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5nurj2sd/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9605417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxyz8gb7
2022-11-23T02:58:46.9606435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxyz8gb7/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9607035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72101i5o
2022-11-23T02:58:46.9607633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72101i5o/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9608194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9608747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9609285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9609806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9610289Z fi_getinfo: -61
2022-11-23T02:58:46.9610646Z fi_getinfo: -61
2022-11-23T02:58:46.9610978Z fi_getinfo: -61
2022-11-23T02:58:46.9611327Z fi_getinfo: -61
2022-11-23T02:58:46.9611626Z ok (6.909s)
2022-11-23T02:58:46.9611806Z 
2022-11-23T02:58:46.9612108Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9612493Z Ran 1 test in 6.909s
2022-11-23T02:58:46.9612690Z 
2022-11-23T02:58:46.9612819Z OK
2022-11-23T02:58:46.9612990Z 
2022-11-23T02:58:46.9613150Z Generating XML reports...
2022-11-23T02:58:46.9613847Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20221123024126.xml
2022-11-23T02:58:46.9614766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9615298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9615949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9616485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9617021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1kedbl0t
2022-11-23T02:58:46.9617629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1kedbl0t/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9617970Z 
2022-11-23T02:58:46.9618101Z Running tests...
2022-11-23T02:58:46.9618578Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9619224Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9619851Z   test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9620409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84447
2022-11-23T02:58:46.9620928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84448
2022-11-23T02:58:46.9621616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9622126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9622775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9623314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9623962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9624470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9625112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9625647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9626167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxldmaqo
2022-11-23T02:58:46.9626861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxldmaqo/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9627470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2a9f6br
2022-11-23T02:58:46.9628081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2a9f6br/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9628645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9629192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9629659Z skip: Need at least 4 CUDA devices (4.339s)
2022-11-23T02:58:46.9629890Z 
2022-11-23T02:58:46.9630200Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9630586Z Ran 1 test in 4.340s
2022-11-23T02:58:46.9630783Z 
2022-11-23T02:58:46.9630927Z OK (skipped=1)
2022-11-23T02:58:46.9631126Z 
2022-11-23T02:58:46.9631287Z Generating XML reports...
2022-11-23T02:58:46.9631999Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024136.xml
2022-11-23T02:58:46.9632879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9633408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9634055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9634638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9635183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_jzwtxx
2022-11-23T02:58:46.9635793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_jzwtxx/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9636134Z 
2022-11-23T02:58:46.9636266Z Running tests...
2022-11-23T02:58:46.9636747Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9637392Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9638026Z   test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9638590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84550
2022-11-23T02:58:46.9639112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84551
2022-11-23T02:58:46.9639800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9640328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9640960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9641506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9642155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9642658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9643303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9643838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9644379Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl74ydy4q
2022-11-23T02:58:46.9644980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl74ydy4q/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9645588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnivh9_d
2022-11-23T02:58:46.9646437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnivh9_d/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9647088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9647633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9648094Z skip: Need at least 4 CUDA devices (4.202s)
2022-11-23T02:58:46.9648322Z 
2022-11-23T02:58:46.9648634Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9649022Z Ran 1 test in 4.202s
2022-11-23T02:58:46.9649221Z 
2022-11-23T02:58:46.9649368Z OK (skipped=1)
2022-11-23T02:58:46.9649561Z 
2022-11-23T02:58:46.9649726Z Generating XML reports...
2022-11-23T02:58:46.9650445Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024143.xml
2022-11-23T02:58:46.9651277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9651806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9652451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9652982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9653517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpce_kds75
2022-11-23T02:58:46.9654122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpce_kds75/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9654458Z 
2022-11-23T02:58:46.9654666Z Running tests...
2022-11-23T02:58:46.9655140Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9655783Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9656405Z   test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9656967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84653
2022-11-23T02:58:46.9657488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84654
2022-11-23T02:58:46.9658170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9658694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9659325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9659870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9660522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9661028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9661671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9662209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9662741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_rq_htj
2022-11-23T02:58:46.9663337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_rq_htj/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9663938Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_xjw2xj
2022-11-23T02:58:46.9664542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_xjw2xj/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9665120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9665650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9666112Z skip: Need at least 4 CUDA devices (4.347s)
2022-11-23T02:58:46.9666448Z 
2022-11-23T02:58:46.9666755Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9667142Z Ran 1 test in 4.347s
2022-11-23T02:58:46.9667342Z 
2022-11-23T02:58:46.9667487Z OK (skipped=1)
2022-11-23T02:58:46.9667682Z 
2022-11-23T02:58:46.9667843Z Generating XML reports...
2022-11-23T02:58:46.9668557Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024149.xml
2022-11-23T02:58:46.9669383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9669911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9670558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9671086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9671629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpipceng99
2022-11-23T02:58:46.9672241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpipceng99/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9672581Z 
2022-11-23T02:58:46.9672728Z Running tests...
2022-11-23T02:58:46.9673187Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9673833Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9674517Z   test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9675096Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84756
2022-11-23T02:58:46.9675616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84757
2022-11-23T02:58:46.9676291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9676818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9677456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9677997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9678648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9679165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9679795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9680332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9680868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbwjlr_fc
2022-11-23T02:58:46.9681470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbwjlr_fc/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9682085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz54snfsy
2022-11-23T02:58:46.9682694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz54snfsy/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9683274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9683804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9684266Z skip: Need at least 4 CUDA devices (4.304s)
2022-11-23T02:58:46.9684495Z 
2022-11-23T02:58:46.9684808Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9685196Z Ran 1 test in 4.304s
2022-11-23T02:58:46.9685394Z 
2022-11-23T02:58:46.9685542Z OK (skipped=1)
2022-11-23T02:58:46.9685956Z 
2022-11-23T02:58:46.9686122Z Generating XML reports...
2022-11-23T02:58:46.9686947Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024156.xml
2022-11-23T02:58:46.9687756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9688276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9688923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9689452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9689995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ubp6mg1
2022-11-23T02:58:46.9690612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ubp6mg1/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9690949Z 
2022-11-23T02:58:46.9691096Z Running tests...
2022-11-23T02:58:46.9691562Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9692216Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9692838Z   test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9693413Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84859
2022-11-23T02:58:46.9693920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84860
2022-11-23T02:58:46.9694600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9695181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9695830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9696369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9697022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9697544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9698173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9698706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9699239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8udx8fzz
2022-11-23T02:58:46.9699842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8udx8fzz/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9700450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplp6w36zw
2022-11-23T02:58:46.9701062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplp6w36zw/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9701645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9702185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9702647Z skip: Need at least 4 CUDA devices (4.337s)
2022-11-23T02:58:46.9702876Z 
2022-11-23T02:58:46.9703183Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9703571Z Ran 1 test in 4.337s
2022-11-23T02:58:46.9703772Z 
2022-11-23T02:58:46.9703918Z OK (skipped=1)
2022-11-23T02:58:46.9704108Z 
2022-11-23T02:58:46.9704269Z Generating XML reports...
2022-11-23T02:58:46.9704999Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024204.xml
2022-11-23T02:58:46.9705812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9706333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9707042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9707588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9708110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90vdn3j_
2022-11-23T02:58:46.9708724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90vdn3j_/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9709061Z 
2022-11-23T02:58:46.9709209Z Running tests...
2022-11-23T02:58:46.9709672Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9710325Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9710958Z   test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9711537Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84962
2022-11-23T02:58:46.9712047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84963
2022-11-23T02:58:46.9712720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9713239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9713871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9714416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9715112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9715637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9716268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9716807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9717343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwwwc2d4r
2022-11-23T02:58:46.9717961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwwwc2d4r/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9718551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ohlgau0
2022-11-23T02:58:46.9719155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ohlgau0/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9719736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9720270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9720731Z skip: Need at least 4 CUDA devices (4.342s)
2022-11-23T02:58:46.9720959Z 
2022-11-23T02:58:46.9721265Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9721671Z Ran 1 test in 4.342s
2022-11-23T02:58:46.9721854Z 
2022-11-23T02:58:46.9722000Z OK (skipped=1)
2022-11-23T02:58:46.9722192Z 
2022-11-23T02:58:46.9722351Z Generating XML reports...
2022-11-23T02:58:46.9723078Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024211.xml
2022-11-23T02:58:46.9723887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9724412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9725061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9725607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9726357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnju5zv8
2022-11-23T02:58:46.9727062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnju5zv8/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9727399Z 
2022-11-23T02:58:46.9727544Z Running tests...
2022-11-23T02:58:46.9728017Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9728668Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9729293Z   test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9729861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85065
2022-11-23T02:58:46.9730374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85066
2022-11-23T02:58:46.9731048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9731569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9732224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9732790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9733442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9733960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9734587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9735192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9735740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5b525p4
2022-11-23T02:58:46.9736358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5b525p4/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9736949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr97o87y3
2022-11-23T02:58:46.9737558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr97o87y3/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9738134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9738665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9739131Z skip: Need at least 4 CUDA devices (4.332s)
2022-11-23T02:58:46.9739361Z 
2022-11-23T02:58:46.9739668Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9740070Z Ran 1 test in 4.332s
2022-11-23T02:58:46.9740253Z 
2022-11-23T02:58:46.9740399Z OK (skipped=1)
2022-11-23T02:58:46.9740588Z 
2022-11-23T02:58:46.9740747Z Generating XML reports...
2022-11-23T02:58:46.9741471Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024218.xml
2022-11-23T02:58:46.9742281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9742807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9743451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9743995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9744516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfo953r43
2022-11-23T02:58:46.9745130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfo953r43/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9745467Z 
2022-11-23T02:58:46.9745614Z Running tests...
2022-11-23T02:58:46.9746092Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9746722Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9747417Z   test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9747994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85168
2022-11-23T02:58:46.9748502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85169
2022-11-23T02:58:46.9749176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9749704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9750357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9750887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9751541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9752065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9752697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9753241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9753779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp06p6j9cb
2022-11-23T02:58:46.9754390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp06p6j9cb/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9755033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7fx03us
2022-11-23T02:58:46.9755646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7fx03us/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9756221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9756769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9757226Z skip: Need at least 4 CUDA devices (4.240s)
2022-11-23T02:58:46.9757455Z 
2022-11-23T02:58:46.9757762Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9758161Z Ran 1 test in 4.240s
2022-11-23T02:58:46.9758358Z 
2022-11-23T02:58:46.9758491Z OK (skipped=1)
2022-11-23T02:58:46.9758682Z 
2022-11-23T02:58:46.9758840Z Generating XML reports...
2022-11-23T02:58:46.9759568Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024225.xml
2022-11-23T02:58:46.9760397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9760909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9761558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9762103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9762626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn07nln_8
2022-11-23T02:58:46.9763237Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn07nln_8/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9763574Z 
2022-11-23T02:58:46.9763722Z Running tests...
2022-11-23T02:58:46.9764198Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9764834Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9765514Z   test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9766351Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85271
2022-11-23T02:58:46.9766857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85272
2022-11-23T02:58:46.9767462Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85273
2022-11-23T02:58:46.9767971Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85274
2022-11-23T02:58:46.9768657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9769165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9769805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9770327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9770975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9771503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9772165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9772705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9773343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9773862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9774507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9775114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9775765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9776286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9776925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9777455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9777997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86jpc4b2
2022-11-23T02:58:46.9778608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86jpc4b2/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9779212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuoffbpbe
2022-11-23T02:58:46.9779808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuoffbpbe/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9780417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmuw67et
2022-11-23T02:58:46.9781020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmuw67et/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9781620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxolhxbkj
2022-11-23T02:58:46.9782210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxolhxbkj/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9782792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9783336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9783854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9784393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9784856Z fi_getinfo: -61
2022-11-23T02:58:46.9785212Z fi_getinfo: -61
2022-11-23T02:58:46.9785545Z fi_getinfo: -61
2022-11-23T02:58:46.9785890Z fi_getinfo: -61
2022-11-23T02:58:46.9786197Z ok (7.811s)
2022-11-23T02:58:46.9786363Z 
2022-11-23T02:58:46.9786668Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9787067Z Ran 1 test in 7.811s
2022-11-23T02:58:46.9787323Z 
2022-11-23T02:58:46.9787451Z OK
2022-11-23T02:58:46.9787470Z 
2022-11-23T02:58:46.9787633Z Generating XML reports...
2022-11-23T02:58:46.9788211Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024231.xml
2022-11-23T02:58:46.9788620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9788833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9789252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9789482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9789772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzv_kitqr
2022-11-23T02:58:46.9790077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzv_kitqr/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9790100Z 
2022-11-23T02:58:46.9790248Z Running tests...
2022-11-23T02:58:46.9790535Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9790931Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9791284Z   test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9791539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85618
2022-11-23T02:58:46.9791832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85619
2022-11-23T02:58:46.9792086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85620
2022-11-23T02:58:46.9792328Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85621
2022-11-23T02:58:46.9792744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9792960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9793364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9793593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9793997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9794206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9794622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9794851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9795250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9795465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9795863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9796093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9796495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9796705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9797119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9797347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9797639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaq7nn_z1
2022-11-23T02:58:46.9798016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaq7nn_z1/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9798301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_y6_dsmf
2022-11-23T02:58:46.9798583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_y6_dsmf/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9798866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbb7dtxcf
2022-11-23T02:58:46.9799163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbb7dtxcf/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9799449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95jjkq04
2022-11-23T02:58:46.9799749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95jjkq04/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9800015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9800280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9800544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9800805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9800979Z fi_getinfo: -61
2022-11-23T02:58:46.9801153Z fi_getinfo: -61
2022-11-23T02:58:46.9801328Z fi_getinfo: -61
2022-11-23T02:58:46.9801501Z fi_getinfo: -61
2022-11-23T02:58:46.9801638Z ok (10.452s)
2022-11-23T02:58:46.9801659Z 
2022-11-23T02:58:46.9801960Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9802160Z Ran 1 test in 10.452s
2022-11-23T02:58:46.9802181Z 
2022-11-23T02:58:46.9802296Z OK
2022-11-23T02:58:46.9802314Z 
2022-11-23T02:58:46.9802477Z Generating XML reports...
2022-11-23T02:58:46.9803067Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024242.xml
2022-11-23T02:58:46.9803478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9803689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9804105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9804332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9804624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqh2gs9as
2022-11-23T02:58:46.9804915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqh2gs9as/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9804951Z 
2022-11-23T02:58:46.9805082Z Running tests...
2022-11-23T02:58:46.9805381Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9805994Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9806375Z   test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9806633Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85965
2022-11-23T02:58:46.9806883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85966
2022-11-23T02:58:46.9807125Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85967
2022-11-23T02:58:46.9807367Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85968
2022-11-23T02:58:46.9807770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9807984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9808404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9808712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9809116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9809326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9809733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9809961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9810352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9810566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9810980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9811207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9811612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9811822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9812237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9812465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9812758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqtx4pfq
2022-11-23T02:58:46.9813117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqtx4pfq/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9813415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvatgezki
2022-11-23T02:58:46.9813704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79bvm5n4
2022-11-23T02:58:46.9814012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvatgezki/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9814311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79bvm5n4/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9814598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8y1mxczd
2022-11-23T02:58:46.9814895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8y1mxczd/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9815162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9815422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9815669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9815922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9816067Z ok (9.832s)
2022-11-23T02:58:46.9816087Z 
2022-11-23T02:58:46.9816392Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9816541Z Ran 1 test in 9.832s
2022-11-23T02:58:46.9816560Z 
2022-11-23T02:58:46.9816688Z OK
2022-11-23T02:58:46.9816707Z 
2022-11-23T02:58:46.9816866Z Generating XML reports...
2022-11-23T02:58:46.9817450Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024255.xml
2022-11-23T02:58:46.9817848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9818061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9818480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9818709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9819060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7nkq3w1
2022-11-23T02:58:46.9819364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7nkq3w1/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9819384Z 
2022-11-23T02:58:46.9819531Z Running tests...
2022-11-23T02:58:46.9819830Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9820226Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9820584Z   test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9820841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86144
2022-11-23T02:58:46.9821087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86145
2022-11-23T02:58:46.9821330Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86146
2022-11-23T02:58:46.9821577Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86147
2022-11-23T02:58:46.9821988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9822201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9822612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9822821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9823276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9823496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9823914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9824145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9824549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9824760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9825174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9825402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9825793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9826004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9826418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9826646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9826942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbemidq46
2022-11-23T02:58:46.9827251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbemidq46/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9827539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifcqz__g
2022-11-23T02:58:46.9827836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifcqz__g/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9828102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_9mitm02
2022-11-23T02:58:46.9828399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_9mitm02/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9828680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4r0bovb
2022-11-23T02:58:46.9828974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4r0bovb/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9829294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9829555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9829811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9830065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9830206Z ok (9.107s)
2022-11-23T02:58:46.9830226Z 
2022-11-23T02:58:46.9830515Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9830671Z Ran 1 test in 9.107s
2022-11-23T02:58:46.9830690Z 
2022-11-23T02:58:46.9830820Z OK
2022-11-23T02:58:46.9830839Z 
2022-11-23T02:58:46.9831003Z Generating XML reports...
2022-11-23T02:58:46.9831591Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024308.xml
2022-11-23T02:58:46.9832004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9832215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9832631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9832884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9833177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8dngerqj
2022-11-23T02:58:46.9833532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8dngerqj/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9833554Z 
2022-11-23T02:58:46.9833705Z Running tests...
2022-11-23T02:58:46.9834008Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9834405Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9834766Z   test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9835022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86379
2022-11-23T02:58:46.9835272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86380
2022-11-23T02:58:46.9835501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86381
2022-11-23T02:58:46.9835745Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86382
2022-11-23T02:58:46.9836158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9836370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9836786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9837019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9837426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9837639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9838037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9838264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9838672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9838882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9839297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9839584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9839991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9840202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9840616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9840826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9841125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1dj0fwv
2022-11-23T02:58:46.9841434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1dj0fwv/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9841719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl69b_q86
2022-11-23T02:58:46.9842014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl69b_q86/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9842301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6x8qadbn
2022-11-23T02:58:46.9842596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6x8qadbn/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9878446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5ylz4_4
2022-11-23T02:58:46.9878815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5ylz4_4/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9879100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9879530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9879816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9880072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9880207Z ok (7.539s)
2022-11-23T02:58:46.9880230Z 
2022-11-23T02:58:46.9880567Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9880708Z Ran 1 test in 7.540s
2022-11-23T02:58:46.9880729Z 
2022-11-23T02:58:46.9880841Z OK
2022-11-23T02:58:46.9880868Z 
2022-11-23T02:58:46.9881016Z Generating XML reports...
2022-11-23T02:58:46.9881610Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024319.xml
2022-11-23T02:58:46.9882027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9882240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9882664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9882897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9883197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdg5cfa6q
2022-11-23T02:58:46.9883511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdg5cfa6q/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9883531Z 
2022-11-23T02:58:46.9883661Z Running tests...
2022-11-23T02:58:46.9883970Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9884370Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9884772Z   test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9885029Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86554
2022-11-23T02:58:46.9885284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86555
2022-11-23T02:58:46.9885626Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86556
2022-11-23T02:58:46.9886092Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86557
2022-11-23T02:58:46.9886528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9886727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9887149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9887384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9887797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9888013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9888436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9888669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9889073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9889268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9889685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9889914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9890410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9890633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9891051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9891281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9891580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_3padd7
2022-11-23T02:58:46.9891889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_3padd7/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9892164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmr_i3x_y
2022-11-23T02:58:46.9892475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmr_i3x_y/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9892771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnize0lri
2022-11-23T02:58:46.9893079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnize0lri/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9893369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiqf272ej
2022-11-23T02:58:46.9893671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiqf272ej/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9893943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9894205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9894469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9894717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9894858Z ok (9.150s)
2022-11-23T02:58:46.9894879Z 
2022-11-23T02:58:46.9895194Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9895346Z Ran 1 test in 9.150s
2022-11-23T02:58:46.9895367Z 
2022-11-23T02:58:46.9895496Z OK
2022-11-23T02:58:46.9895514Z 
2022-11-23T02:58:46.9895677Z Generating XML reports...
2022-11-23T02:58:46.9896271Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024330.xml
2022-11-23T02:58:46.9896783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9896983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9897403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9897636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9897934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphmbv3dw9
2022-11-23T02:58:46.9898242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphmbv3dw9/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9898261Z 
2022-11-23T02:58:46.9898410Z Running tests...
2022-11-23T02:58:46.9898714Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9899168Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9899558Z   test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9899798Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86793
2022-11-23T02:58:46.9900058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86794
2022-11-23T02:58:46.9900313Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86795
2022-11-23T02:58:46.9900622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86796
2022-11-23T02:58:46.9901046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9901261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9901681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9901916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9902308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9902523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9902936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9903164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9903572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9903787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9904194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9904423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9904823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9905018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9905435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9905661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9905959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpctg6ea42
2022-11-23T02:58:46.9906265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpctg6ea42/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9906555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsaa1fumi
2022-11-23T02:58:46.9906921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsaa1fumi/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9907216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsfc012ps
2022-11-23T02:58:46.9907504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsfc012ps/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9907798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95vht434
2022-11-23T02:58:46.9908102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95vht434/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9908375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9908637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9908905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9909167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9909313Z ok (9.253s)
2022-11-23T02:58:46.9909334Z 
2022-11-23T02:58:46.9909646Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9909781Z Ran 1 test in 9.253s
2022-11-23T02:58:46.9909800Z 
2022-11-23T02:58:46.9909931Z OK
2022-11-23T02:58:46.9909950Z 
2022-11-23T02:58:46.9910113Z Generating XML reports...
2022-11-23T02:58:46.9910704Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024342.xml
2022-11-23T02:58:46.9911197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9911422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9911842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9912079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9912356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63nth5qe
2022-11-23T02:58:46.9912666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63nth5qe/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9912686Z 
2022-11-23T02:58:46.9912834Z Running tests...
2022-11-23T02:58:46.9913134Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9913533Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9913922Z   test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9914181Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86972
2022-11-23T02:58:46.9914437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86973
2022-11-23T02:58:46.9914695Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86974
2022-11-23T02:58:46.9914930Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86975
2022-11-23T02:58:46.9915347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9915562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9915983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9916217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9916624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9916837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9917248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9917533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9917921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9918135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9918546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9918773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9919175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9919387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9919805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9920034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9920310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifnflwjw
2022-11-23T02:58:46.9920625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifnflwjw/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9920918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyw3ozvxn
2022-11-23T02:58:46.9921225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyw3ozvxn/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9921565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zqqo6wg
2022-11-23T02:58:46.9921878Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zqqo6wg/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9922169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu86gxzcg
2022-11-23T02:58:46.9922477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu86gxzcg/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9922741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9922983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9923247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9923512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9923653Z ok (9.123s)
2022-11-23T02:58:46.9923673Z 
2022-11-23T02:58:46.9923986Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9924139Z Ran 1 test in 9.123s
2022-11-23T02:58:46.9924158Z 
2022-11-23T02:58:46.9924288Z OK
2022-11-23T02:58:46.9924307Z 
2022-11-23T02:58:46.9924469Z Generating XML reports...
2022-11-23T02:58:46.9925041Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024354.xml
2022-11-23T02:58:46.9925460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9925919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9926359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9926591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9926892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdc7xma2
2022-11-23T02:58:46.9927203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdc7xma2/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9927223Z 
2022-11-23T02:58:46.9927373Z Running tests...
2022-11-23T02:58:46.9927676Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9928144Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9928516Z   test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9928773Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87207
2022-11-23T02:58:46.9929031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87208
2022-11-23T02:58:46.9929286Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87209
2022-11-23T02:58:46.9929544Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87210
2022-11-23T02:58:46.9929961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9930175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9930584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9930818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9931228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9931441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9931852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9932136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9932552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9932803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9933217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9933429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9933832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9934044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9934460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9934686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9934986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgm2gmbjc
2022-11-23T02:58:46.9935296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgm2gmbjc/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9935587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4iwc7o1t
2022-11-23T02:58:46.9935880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4iwc7o1t/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9936176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0hk3mng
2022-11-23T02:58:46.9936479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0hk3mng/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9936771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu70ct_as
2022-11-23T02:58:46.9937076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu70ct_as/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9937348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9937610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9937876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9938197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9938320Z ok (8.714s)
2022-11-23T02:58:46.9938341Z 
2022-11-23T02:58:46.9938649Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9938800Z Ran 1 test in 8.715s
2022-11-23T02:58:46.9938819Z 
2022-11-23T02:58:46.9938950Z OK
2022-11-23T02:58:46.9938969Z 
2022-11-23T02:58:46.9939132Z Generating XML reports...
2022-11-23T02:58:46.9939725Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024405.xml
2022-11-23T02:58:46.9940141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9940356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9940775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9940994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9941287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqvi9znul
2022-11-23T02:58:46.9941597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqvi9znul/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9941617Z 
2022-11-23T02:58:46.9941765Z Running tests...
2022-11-23T02:58:46.9942066Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9942512Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9942873Z   test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9943134Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87382
2022-11-23T02:58:46.9943371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87383
2022-11-23T02:58:46.9943635Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87384
2022-11-23T02:58:46.9943891Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87385
2022-11-23T02:58:46.9944303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9944517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9944936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9945169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9945580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9945794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9946194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9946425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9946827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9947041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9947457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9947689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9948091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9948306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9948702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9948986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9949280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ho7_cip
2022-11-23T02:58:46.9949585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ho7_cip/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9949880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_h8u6_45
2022-11-23T02:58:46.9950182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_h8u6_45/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9950477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3101a_cd
2022-11-23T02:58:46.9950780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3101a_cd/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9951073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdb0_917m
2022-11-23T02:58:46.9951361Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdb0_917m/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9951626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9951885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9952147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9952410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9952552Z ok (4.429s)
2022-11-23T02:58:46.9952616Z 
2022-11-23T02:58:46.9952931Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9953083Z Ran 1 test in 4.430s
2022-11-23T02:58:46.9953102Z 
2022-11-23T02:58:46.9953215Z OK
2022-11-23T02:58:46.9953233Z 
2022-11-23T02:58:46.9953399Z Generating XML reports...
2022-11-23T02:58:46.9953997Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024417.xml
2022-11-23T02:58:46.9954407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9954624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9955044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9955277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9955573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppc0fhcb9
2022-11-23T02:58:46.9955880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppc0fhcb9/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9955900Z 
2022-11-23T02:58:46.9956031Z Running tests...
2022-11-23T02:58:46.9956336Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9956737Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9957084Z   test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9957340Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87553
2022-11-23T02:58:46.9957597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87554
2022-11-23T02:58:46.9957848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87555
2022-11-23T02:58:46.9958106Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87556
2022-11-23T02:58:46.9958504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9958721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9959199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9959429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9959838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9960052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9960462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9960693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9961096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9961292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9961705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9961936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9962332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9962542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9962957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9963181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9963523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpgiv9do
2022-11-23T02:58:46.9963825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpgiv9do/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9964120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivlks68s
2022-11-23T02:58:46.9964433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivlks68s/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9964726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg4r56ml
2022-11-23T02:58:46.9965031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg4r56ml/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9965321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp948x4ldx
2022-11-23T02:58:46.9965623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp948x4ldx/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9966121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9966388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9966636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9966908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9967051Z ok (4.434s)
2022-11-23T02:58:46.9967071Z 
2022-11-23T02:58:46.9967386Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9967540Z Ran 1 test in 4.434s
2022-11-23T02:58:46.9967559Z 
2022-11-23T02:58:46.9967689Z OK
2022-11-23T02:58:46.9967708Z 
2022-11-23T02:58:46.9967873Z Generating XML reports...
2022-11-23T02:58:46.9968461Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024424.xml
2022-11-23T02:58:46.9968878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9969077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9969497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9969833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9970130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu4rj0dgn
2022-11-23T02:58:46.9970438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu4rj0dgn/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9970457Z 
2022-11-23T02:58:46.9970609Z Running tests...
2022-11-23T02:58:46.9970914Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9971318Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9971645Z   test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9971905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87724
2022-11-23T02:58:46.9972162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87725
2022-11-23T02:58:46.9972420Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87726
2022-11-23T02:58:46.9972675Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87727
2022-11-23T02:58:46.9973086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9973301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9973720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9974015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9974420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9974633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9975050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9975277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9975680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9975895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9976308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9976538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9976921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9977135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9977551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9977781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9978078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi4ux2ok8
2022-11-23T02:58:46.9978389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi4ux2ok8/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9978684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpano2hph_
2022-11-23T02:58:46.9978992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpano2hph_/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9979288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjsar_j3
2022-11-23T02:58:46.9979573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjsar_j3/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9979867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb8mf92_7
2022-11-23T02:58:46.9980226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb8mf92_7/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9980493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9980753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9981019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9981286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9981427Z ok (4.436s)
2022-11-23T02:58:46.9981451Z 
2022-11-23T02:58:46.9981744Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9981897Z Ran 1 test in 4.436s
2022-11-23T02:58:46.9981916Z 
2022-11-23T02:58:46.9982051Z OK
2022-11-23T02:58:46.9982069Z 
2022-11-23T02:58:46.9982232Z Generating XML reports...
2022-11-23T02:58:46.9982829Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024431.xml
2022-11-23T02:58:46.9983238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9983456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9983873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9984104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9984425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3hni6tbs
2022-11-23T02:58:46.9984738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3hni6tbs/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9984758Z 
2022-11-23T02:58:46.9984907Z Running tests...
2022-11-23T02:58:46.9985210Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9985613Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:46.9985961Z   test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:46.9986220Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87895
2022-11-23T02:58:46.9986476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87896
2022-11-23T02:58:46.9986713Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87897
2022-11-23T02:58:46.9986972Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87898
2022-11-23T02:58:46.9987387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9987602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9988028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9988257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9988661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9988874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9989286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9989500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9989905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9990121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9990534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9990809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9991213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9991427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9991841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9992050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9992350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_0j6m8j
2022-11-23T02:58:46.9992656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_0j6m8j/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9992949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74a1x2cc
2022-11-23T02:58:46.9993257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74a1x2cc/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9993551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzpva16qc
2022-11-23T02:58:46.9993862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzpva16qc/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9994154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbrhxq2zz
2022-11-23T02:58:46.9994463Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbrhxq2zz/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9994760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:46.9995030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:46.9995294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:46.9995564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:46.9995706Z ok (4.533s)
2022-11-23T02:58:46.9995726Z 
2022-11-23T02:58:46.9996037Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9996188Z Ran 1 test in 4.533s
2022-11-23T02:58:46.9996208Z 
2022-11-23T02:58:46.9996337Z OK
2022-11-23T02:58:46.9996356Z 
2022-11-23T02:58:46.9996500Z Generating XML reports...
2022-11-23T02:58:46.9997094Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024438.xml
2022-11-23T02:58:46.9997507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:46.9997723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:46.9998140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:46.9998376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:46.9998672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaidkpvyk
2022-11-23T02:58:46.9998984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaidkpvyk/_remote_module_non_scriptable.py
2022-11-23T02:58:46.9999004Z 
2022-11-23T02:58:46.9999154Z Running tests...
2022-11-23T02:58:46.9999437Z ----------------------------------------------------------------------
2022-11-23T02:58:46.9999843Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0000205Z   test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0000462Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88066
2022-11-23T02:58:47.0000721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88067
2022-11-23T02:58:47.0001029Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88068
2022-11-23T02:58:47.0001284Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88069
2022-11-23T02:58:47.0001700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0001896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0002320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0002558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0002965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0003178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0003594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0003821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0004219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0004436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0004829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0005101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0005509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0005935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0006374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0006606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0006902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwjxpqq4
2022-11-23T02:58:47.0007215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwjxpqq4/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0007508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp18ryyc8a
2022-11-23T02:58:47.0007796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp18ryyc8a/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0008093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa04t6ejq
2022-11-23T02:58:47.0008399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa04t6ejq/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0008690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpth0vyldl
2022-11-23T02:58:47.0008999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpth0vyldl/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0009265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0009528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0009793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0010042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0010185Z ok (5.330s)
2022-11-23T02:58:47.0010210Z 
2022-11-23T02:58:47.0010520Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0010674Z Ran 1 test in 5.330s
2022-11-23T02:58:47.0010693Z 
2022-11-23T02:58:47.0010823Z OK
2022-11-23T02:58:47.0010842Z 
2022-11-23T02:58:47.0011005Z Generating XML reports...
2022-11-23T02:58:47.0011677Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024446.xml
2022-11-23T02:58:47.0012093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0012311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0012710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0012944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0013245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8b89i9yn
2022-11-23T02:58:47.0013550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8b89i9yn/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0013570Z 
2022-11-23T02:58:47.0013718Z Running tests...
2022-11-23T02:58:47.0014020Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0014427Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0014777Z   test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0015018Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88241
2022-11-23T02:58:47.0015276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88242
2022-11-23T02:58:47.0015530Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88243
2022-11-23T02:58:47.0015840Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88244
2022-11-23T02:58:47.0016265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0016482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0016907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0017138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0017546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0017741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0018155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0018389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0018791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0019003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0019409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0019638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0020038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0020234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0020653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0020883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0021178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiiespae0
2022-11-23T02:58:47.0021488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiiespae0/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0021779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd27nd4zd
2022-11-23T02:58:47.0022140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd27nd4zd/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0022436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixmkrydf
2022-11-23T02:58:47.0022743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixmkrydf/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0023018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2uqvobw6
2022-11-23T02:58:47.0023321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2uqvobw6/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0023592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0023856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0024120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0024388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0024529Z ok (5.350s)
2022-11-23T02:58:47.0024548Z 
2022-11-23T02:58:47.0024859Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0024992Z Ran 1 test in 5.350s
2022-11-23T02:58:47.0025034Z 
2022-11-23T02:58:47.0025145Z OK
2022-11-23T02:58:47.0025165Z 
2022-11-23T02:58:47.0025329Z Generating XML reports...
2022-11-23T02:58:47.0025919Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024454.xml
2022-11-23T02:58:47.0026375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0026600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0027022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0027258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0027554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn75bdldg
2022-11-23T02:58:47.0027846Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn75bdldg/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0027865Z 
2022-11-23T02:58:47.0028014Z Running tests...
2022-11-23T02:58:47.0028315Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0028716Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0029070Z   test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0029328Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88416
2022-11-23T02:58:47.0029588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88417
2022-11-23T02:58:47.0029844Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88418
2022-11-23T02:58:47.0030080Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88419
2022-11-23T02:58:47.0030494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0030712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0031128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0031360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0031765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0031977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0032446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0032676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0033105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0033326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0033737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0033968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0034369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0034581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0035000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0035230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0035507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxj9x3b1l
2022-11-23T02:58:47.0035817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxj9x3b1l/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0036111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptav53px6
2022-11-23T02:58:47.0036468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptav53px6/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0036771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0800ixs3
2022-11-23T02:58:47.0037076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0800ixs3/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0037369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiifvuahf
2022-11-23T02:58:47.0037681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiifvuahf/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0037948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0038191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0038454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0038719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0038860Z ok (9.561s)
2022-11-23T02:58:47.0038883Z 
2022-11-23T02:58:47.0039190Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0039344Z Ran 1 test in 9.561s
2022-11-23T02:58:47.0039363Z 
2022-11-23T02:58:47.0039495Z OK
2022-11-23T02:58:47.0039513Z 
2022-11-23T02:58:47.0039675Z Generating XML reports...
2022-11-23T02:58:47.0040253Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024502.xml
2022-11-23T02:58:47.0040668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0040885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0041303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0041533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0041830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn2ksd2z4
2022-11-23T02:58:47.0042135Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn2ksd2z4/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0042154Z 
2022-11-23T02:58:47.0042302Z Running tests...
2022-11-23T02:58:47.0042601Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0043045Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0043378Z   test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0044194Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79750 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2022-11-23T02:58:47.0044219Z 
2022-11-23T02:58:47.0044522Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0044674Z Ran 1 test in 1.628s
2022-11-23T02:58:47.0044694Z 
2022-11-23T02:58:47.0044843Z OK (skipped=1)
2022-11-23T02:58:47.0044862Z 
2022-11-23T02:58:47.0045025Z Generating XML reports...
2022-11-23T02:58:47.0045615Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024514.xml
2022-11-23T02:58:47.0046287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0046505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0046907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0047142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0047511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2_bg0kn
2022-11-23T02:58:47.0047830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2_bg0kn/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0047849Z 
2022-11-23T02:58:47.0047998Z Running tests...
2022-11-23T02:58:47.0048302Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0048706Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0049044Z   test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0049304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88629
2022-11-23T02:58:47.0049545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88630
2022-11-23T02:58:47.0049799Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88631
2022-11-23T02:58:47.0050056Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88632
2022-11-23T02:58:47.0050472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0050687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0051111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0051341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0051746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0051944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0052358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0052589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0052992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0053204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0053619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0053914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0054321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0054538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0054937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0055171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0055470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplgo6245c
2022-11-23T02:58:47.0055779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplgo6245c/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0056068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_clm24cl
2022-11-23T02:58:47.0056362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkbwqxazo
2022-11-23T02:58:47.0056666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_clm24cl/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0056966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkbwqxazo/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0057243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgzgp_2j
2022-11-23T02:58:47.0057549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgzgp_2j/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0057872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0058144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0058408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0058676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0058864Z fi_getinfo: -61
2022-11-23T02:58:47.0059042Z fi_getinfo: -61
2022-11-23T02:58:47.0059200Z fi_getinfo: -61
2022-11-23T02:58:47.0059378Z fi_getinfo: -61
2022-11-23T02:58:47.0059515Z ok (18.367s)
2022-11-23T02:58:47.0059535Z 
2022-11-23T02:58:47.0059837Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0059989Z Ran 1 test in 18.367s
2022-11-23T02:58:47.0060008Z 
2022-11-23T02:58:47.0060138Z OK
2022-11-23T02:58:47.0060158Z 
2022-11-23T02:58:47.0060321Z Generating XML reports...
2022-11-23T02:58:47.0060916Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024518.xml
2022-11-23T02:58:47.0061312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0061534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0061952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0062184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0062480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwzlltzje
2022-11-23T02:58:47.0062788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwzlltzje/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0062807Z 
2022-11-23T02:58:47.0062953Z Running tests...
2022-11-23T02:58:47.0063261Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0063660Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0063981Z   test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0064297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88988
2022-11-23T02:58:47.0064556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88989
2022-11-23T02:58:47.0064813Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88990
2022-11-23T02:58:47.0065067Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88991
2022-11-23T02:58:47.0065481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0065696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0066115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0066328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0066740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0066959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0067371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0067598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0068001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0068212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0068662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0068898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0069281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0069500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0069916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0070141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0070438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgszzn5x8
2022-11-23T02:58:47.0070749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgszzn5x8/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0071044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gk4qjv0
2022-11-23T02:58:47.0071349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gk4qjv0/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0071624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkkawrt9
2022-11-23T02:58:47.0071932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkkawrt9/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0072228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpec8rsaga
2022-11-23T02:58:47.0072534Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpec8rsaga/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0072801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0073063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0073330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0073593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0073781Z fi_getinfo: -61
2022-11-23T02:58:47.0073943Z fi_getinfo: -61
2022-11-23T02:58:47.0074119Z fi_getinfo: -61
2022-11-23T02:58:47.0074293Z fi_getinfo: -61
2022-11-23T02:58:47.0074524Z ok (10.652s)
2022-11-23T02:58:47.0074543Z 
2022-11-23T02:58:47.0074848Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0075001Z Ran 1 test in 10.652s
2022-11-23T02:58:47.0075020Z 
2022-11-23T02:58:47.0075153Z OK
2022-11-23T02:58:47.0075172Z 
2022-11-23T02:58:47.0075317Z Generating XML reports...
2022-11-23T02:58:47.0075911Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024539.xml
2022-11-23T02:58:47.0076324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0076541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0076960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0077191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0077487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_ytcdeg
2022-11-23T02:58:47.0077794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_ytcdeg/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0077814Z 
2022-11-23T02:58:47.0077963Z Running tests...
2022-11-23T02:58:47.0078249Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0078647Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0079038Z   test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0079305Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89347
2022-11-23T02:58:47.0079562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89348
2022-11-23T02:58:47.0079816Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89349
2022-11-23T02:58:47.0080072Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89350
2022-11-23T02:58:47.0080488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0080684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0081104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0081334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0081750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0081964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0082380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0082612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0083013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0083225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0083617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0083843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0084245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0084456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0084872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0085098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0085445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6c53jeiy
2022-11-23T02:58:47.0085978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6c53jeiy/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0086263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdowcab65
2022-11-23T02:58:47.0086572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdowcab65/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0086867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzj704o9
2022-11-23T02:58:47.0087175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzj704o9/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0087464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5f6wo_dh
2022-11-23T02:58:47.0087766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5f6wo_dh/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0088037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0088301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0088558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0088804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0089000Z fi_getinfo: -61
2022-11-23T02:58:47.0089179Z fi_getinfo: -61
2022-11-23T02:58:47.0089357Z fi_getinfo: -61
2022-11-23T02:58:47.0089533Z fi_getinfo: -61
2022-11-23T02:58:47.0089747Z ok (9.857s)
2022-11-23T02:58:47.0089770Z 
2022-11-23T02:58:47.0090085Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0090218Z Ran 1 test in 9.857s
2022-11-23T02:58:47.0090237Z 
2022-11-23T02:58:47.0090370Z OK
2022-11-23T02:58:47.0090389Z 
2022-11-23T02:58:47.0090551Z Generating XML reports...
2022-11-23T02:58:47.0091150Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024553.xml
2022-11-23T02:58:47.0091560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0091774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0092193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0092422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0092704Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpjqwkyz
2022-11-23T02:58:47.0093017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpjqwkyz/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0093037Z 
2022-11-23T02:58:47.0093183Z Running tests...
2022-11-23T02:58:47.0093483Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0093881Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0094216Z   test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0094475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89701
2022-11-23T02:58:47.0094732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89702
2022-11-23T02:58:47.0094983Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89703
2022-11-23T02:58:47.0095222Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89704
2022-11-23T02:58:47.0095637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0095851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0096344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0096573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0096982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0097195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0097603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0097816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0098218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0098433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0098843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0099072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0099468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0099679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0100094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0100365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0100650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnphqx8yn
2022-11-23T02:58:47.0100962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnphqx8yn/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0101255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvium9f4r
2022-11-23T02:58:47.0101567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvium9f4r/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0101859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpebk_qky8
2022-11-23T02:58:47.0102160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpebk_qky8/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0102447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfltut66q
2022-11-23T02:58:47.0102751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfltut66q/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0103022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0103267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0103529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0103797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0103985Z fi_getinfo: -61
2022-11-23T02:58:47.0104161Z fi_getinfo: -61
2022-11-23T02:58:47.0104337Z fi_getinfo: -61
2022-11-23T02:58:47.0104516Z fi_getinfo: -61
2022-11-23T02:58:47.0104638Z ok (4.894s)
2022-11-23T02:58:47.0104658Z 
2022-11-23T02:58:47.0104960Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0105111Z Ran 1 test in 4.894s
2022-11-23T02:58:47.0105130Z 
2022-11-23T02:58:47.0105257Z OK
2022-11-23T02:58:47.0105277Z 
2022-11-23T02:58:47.0105439Z Generating XML reports...
2022-11-23T02:58:47.0106035Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024605.xml
2022-11-23T02:58:47.0106447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0106715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0107117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0107352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0107645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zadhi2o
2022-11-23T02:58:47.0107951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zadhi2o/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0107971Z 
2022-11-23T02:58:47.0108119Z Running tests...
2022-11-23T02:58:47.0108426Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0108824Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0109181Z   test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0109442Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90044
2022-11-23T02:58:47.0109681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90045
2022-11-23T02:58:47.0109936Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90046
2022-11-23T02:58:47.0110187Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90047
2022-11-23T02:58:47.0110599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0110862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0111293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0111523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0111929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0112127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0112540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0112768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0113168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0113379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0113791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0114016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0114413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0114629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0115027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0115255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0115552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1uvwef0b
2022-11-23T02:58:47.0115859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1uvwef0b/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0116153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpas4j9m4s
2022-11-23T02:58:47.0116460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpas4j9m4s/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0116752Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyopm40by
2022-11-23T02:58:47.0117130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyopm40by/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0117423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijc4cf17
2022-11-23T02:58:47.0117706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijc4cf17/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0117974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0118231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0118502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0118773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0118960Z fi_getinfo: -61
2022-11-23T02:58:47.0119134Z fi_getinfo: -61
2022-11-23T02:58:47.0119291Z fi_getinfo: -61
2022-11-23T02:58:47.0119465Z fi_getinfo: -61
2022-11-23T02:58:47.0119605Z ok (6.440s)
2022-11-23T02:58:47.0119626Z 
2022-11-23T02:58:47.0119927Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0120076Z Ran 1 test in 6.440s
2022-11-23T02:58:47.0120095Z 
2022-11-23T02:58:47.0120219Z OK
2022-11-23T02:58:47.0120239Z 
2022-11-23T02:58:47.0120397Z Generating XML reports...
2022-11-23T02:58:47.0120987Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024613.xml
2022-11-23T02:58:47.0121427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0121649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0122062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0122293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0122590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1rwt9kp
2022-11-23T02:58:47.0122894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1rwt9kp/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0122914Z 
2022-11-23T02:58:47.0123056Z Running tests...
2022-11-23T02:58:47.0123351Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0123749Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0124096Z   test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0124353Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90395
2022-11-23T02:58:47.0124611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90396
2022-11-23T02:58:47.0124865Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90397
2022-11-23T02:58:47.0125119Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90398
2022-11-23T02:58:47.0125527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0125941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0126372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0126584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0126994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0127208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0127621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0127927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0128334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0128547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0128957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0129180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0129562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0129777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0130192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0130410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0130705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgg_064vl
2022-11-23T02:58:47.0131007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgg_064vl/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0131293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnm5snngl
2022-11-23T02:58:47.0131597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnm5snngl/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0131866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmm0ct7b
2022-11-23T02:58:47.0132232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmm0ct7b/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0132524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgidftdbx
2022-11-23T02:58:47.0132862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgidftdbx/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0133133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0133391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0133655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0133917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0134100Z fi_getinfo: -61
2022-11-23T02:58:47.0134258Z fi_getinfo: -61
2022-11-23T02:58:47.0134433Z fi_getinfo: -61
2022-11-23T02:58:47.0134612Z fi_getinfo: -61
2022-11-23T02:58:47.0134750Z ok (6.447s)
2022-11-23T02:58:47.0134769Z 
2022-11-23T02:58:47.0135066Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0135215Z Ran 1 test in 6.447s
2022-11-23T02:58:47.0135234Z 
2022-11-23T02:58:47.0135360Z OK
2022-11-23T02:58:47.0135381Z 
2022-11-23T02:58:47.0135530Z Generating XML reports...
2022-11-23T02:58:47.0136119Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024622.xml
2022-11-23T02:58:47.0136532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0136744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0137161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0137396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0137689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgc9n76fp
2022-11-23T02:58:47.0137994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgc9n76fp/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0138014Z 
2022-11-23T02:58:47.0138159Z Running tests...
2022-11-23T02:58:47.0138503Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0138905Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0139248Z   test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0139505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90746
2022-11-23T02:58:47.0139762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90747
2022-11-23T02:58:47.0140021Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90748
2022-11-23T02:58:47.0140274Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90749
2022-11-23T02:58:47.0140679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0140877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0141300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0141523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0141926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0142136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0142545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0142815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0143223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0143434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0143828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0144049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0144443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0144651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0145062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0145285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0145572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1x8q1n8j
2022-11-23T02:58:47.0145873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1x8q1n8j/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0146146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9u4j3jrj
2022-11-23T02:58:47.0146451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9u4j3jrj/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0146738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15rz4wmf
2022-11-23T02:58:47.0147038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15rz4wmf/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0147331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnemcd32b
2022-11-23T02:58:47.0147637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnemcd32b/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0147902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0148159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0148418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0148721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0148905Z fi_getinfo: -61
2022-11-23T02:58:47.0149079Z fi_getinfo: -61
2022-11-23T02:58:47.0149252Z fi_getinfo: -61
2022-11-23T02:58:47.0149421Z fi_getinfo: -61
2022-11-23T02:58:47.0149556Z ok (6.431s)
2022-11-23T02:58:47.0149576Z 
2022-11-23T02:58:47.0149874Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0150007Z Ran 1 test in 6.431s
2022-11-23T02:58:47.0150027Z 
2022-11-23T02:58:47.0150153Z OK
2022-11-23T02:58:47.0150172Z 
2022-11-23T02:58:47.0150334Z Generating XML reports...
2022-11-23T02:58:47.0150927Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024631.xml
2022-11-23T02:58:47.0151335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0151553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0151970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0152201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0152490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3jrfs9m
2022-11-23T02:58:47.0152780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3jrfs9m/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0152800Z 
2022-11-23T02:58:47.0152992Z Running tests...
2022-11-23T02:58:47.0153296Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0153694Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0154059Z   test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0154863Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/80008 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.615s)
2022-11-23T02:58:47.0154884Z 
2022-11-23T02:58:47.0155180Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0155328Z Ran 1 test in 1.615s
2022-11-23T02:58:47.0155348Z 
2022-11-23T02:58:47.0155490Z OK (skipped=1)
2022-11-23T02:58:47.0155513Z 
2022-11-23T02:58:47.0155678Z Generating XML reports...
2022-11-23T02:58:47.0156244Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024640.xml
2022-11-23T02:58:47.0156650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0156867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0157283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0157513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0157802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7isd8dd
2022-11-23T02:58:47.0158107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7isd8dd/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0158128Z 
2022-11-23T02:58:47.0158278Z Running tests...
2022-11-23T02:58:47.0158564Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0158961Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0159300Z   test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0159607Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91127
2022-11-23T02:58:47.0159863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91128
2022-11-23T02:58:47.0160113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91129
2022-11-23T02:58:47.0160363Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91130
2022-11-23T02:58:47.0160770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0160986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0161391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0161620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0162030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0162237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0162647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0162870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0163281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0163537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0163940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0164166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0164566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0164786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0165193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0165412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0165905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsjn802hv
2022-11-23T02:58:47.0166222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsjn802hv/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0166518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal1vjhsn
2022-11-23T02:58:47.0166811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal1vjhsn/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0167101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptw966t5l
2022-11-23T02:58:47.0167412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptw966t5l/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0167697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5eiu8j6j
2022-11-23T02:58:47.0167995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5eiu8j6j/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0168257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0168516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0168785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0169032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0169222Z fi_getinfo: -61
2022-11-23T02:58:47.0169397Z fi_getinfo: -61
2022-11-23T02:58:47.0169569Z fi_getinfo: -61
2022-11-23T02:58:47.0169822Z fi_getinfo: -61
2022-11-23T02:58:47.0169959Z ok (7.735s)
2022-11-23T02:58:47.0169980Z 
2022-11-23T02:58:47.0170282Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0170415Z Ran 1 test in 7.735s
2022-11-23T02:58:47.0170452Z 
2022-11-23T02:58:47.0170565Z OK
2022-11-23T02:58:47.0170583Z 
2022-11-23T02:58:47.0170747Z Generating XML reports...
2022-11-23T02:58:47.0171336Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024645.xml
2022-11-23T02:58:47.0171751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0171961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0172380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0172607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0172906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaxv5m9g5
2022-11-23T02:58:47.0173199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaxv5m9g5/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0173235Z 
2022-11-23T02:58:47.0173368Z Running tests...
2022-11-23T02:58:47.0173670Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0174070Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0174470Z   test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0174737Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91478
2022-11-23T02:58:47.0174991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91479
2022-11-23T02:58:47.0175245Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91480
2022-11-23T02:58:47.0175497Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91481
2022-11-23T02:58:47.0175895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0176111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0176528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0176752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0177159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0177366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0177777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0178009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0178392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0178604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0179014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0179238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0179649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0179858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0180266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0180552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0180851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvepwt029
2022-11-23T02:58:47.0181141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvepwt029/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0181438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk8ttr5s4
2022-11-23T02:58:47.0181745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk8ttr5s4/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0182039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4bqv4r9b
2022-11-23T02:58:47.0182343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4bqv4r9b/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0182631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqw619e48
2022-11-23T02:58:47.0182931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqw619e48/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0183202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0183450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0183714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0183979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0184168Z fi_getinfo: -61
2022-11-23T02:58:47.0184346Z fi_getinfo: -61
2022-11-23T02:58:47.0184522Z fi_getinfo: -61
2022-11-23T02:58:47.0184743Z fi_getinfo: -61
2022-11-23T02:58:47.0184870Z ok (7.637s)
2022-11-23T02:58:47.0184908Z 
2022-11-23T02:58:47.0185194Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0185346Z Ran 1 test in 7.637s
2022-11-23T02:58:47.0185366Z 
2022-11-23T02:58:47.0185493Z OK
2022-11-23T02:58:47.0185516Z 
2022-11-23T02:58:47.0185677Z Generating XML reports...
2022-11-23T02:58:47.0186264Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024655.xml
2022-11-23T02:58:47.0186670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0186883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0187302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0187520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0187815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzb2ifpbg
2022-11-23T02:58:47.0188120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzb2ifpbg/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0188143Z 
2022-11-23T02:58:47.0188288Z Running tests...
2022-11-23T02:58:47.0188590Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0188985Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0189326Z   test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0189577Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91829
2022-11-23T02:58:47.0189832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91830
2022-11-23T02:58:47.0190072Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91831
2022-11-23T02:58:47.0190321Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91832
2022-11-23T02:58:47.0190733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0191018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0191437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0191664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0192067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0192274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0192671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0192901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0193296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0193506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0193916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0194139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0194540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0194746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0195152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0195407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0195710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkz5y0j3_
2022-11-23T02:58:47.0196016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkz5y0j3_/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0196313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwi8br1ym
2022-11-23T02:58:47.0196619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwi8br1ym/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0196911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8y8majuc
2022-11-23T02:58:47.0197213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8y8majuc/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0197501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2oigqm8
2022-11-23T02:58:47.0197790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2oigqm8/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0198055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0198320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0198582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0198841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0199027Z fi_getinfo: -61
2022-11-23T02:58:47.0199258Z fi_getinfo: -61
2022-11-23T02:58:47.0199433Z fi_getinfo: -61
2022-11-23T02:58:47.0199590Z fi_getinfo: -61
2022-11-23T02:58:47.0199728Z ok (7.619s)
2022-11-23T02:58:47.0199747Z 
2022-11-23T02:58:47.0200046Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0200197Z Ran 1 test in 7.619s
2022-11-23T02:58:47.0200216Z 
2022-11-23T02:58:47.0200343Z OK
2022-11-23T02:58:47.0200366Z 
2022-11-23T02:58:47.0200523Z Generating XML reports...
2022-11-23T02:58:47.0201114Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024705.xml
2022-11-23T02:58:47.0201522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0201777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0202202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0202428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0202722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpro0051bu
2022-11-23T02:58:47.0203023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpro0051bu/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0203046Z 
2022-11-23T02:58:47.0203193Z Running tests...
2022-11-23T02:58:47.0203494Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0203892Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0204234Z   test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0204480Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92180
2022-11-23T02:58:47.0204736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92181
2022-11-23T02:58:47.0204980Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92182
2022-11-23T02:58:47.0205228Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92183
2022-11-23T02:58:47.0205638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0206130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0206571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0206799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0207190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0207402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0207813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0208041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0208437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0208650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0209056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0209282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0209676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0209873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0210280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0210505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0210801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjm4kkd54
2022-11-23T02:58:47.0211109Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjm4kkd54/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0211398Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz3w_x0kz
2022-11-23T02:58:47.0211703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz3w_x0kz/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0211991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcy5224w
2022-11-23T02:58:47.0212342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcy5224w/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0212633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfk6pyama
2022-11-23T02:58:47.0212943Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfk6pyama/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0213212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0213479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0213748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0213938Z fi_getinfo: -61
2022-11-23T02:58:47.0214111Z fi_getinfo: -61
2022-11-23T02:58:47.0214356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0214538Z fi_getinfo: -61
2022-11-23T02:58:47.0214718Z fi_getinfo: -61
2022-11-23T02:58:47.0214856Z ok (7.725s)
2022-11-23T02:58:47.0214876Z 
2022-11-23T02:58:47.0215180Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0215332Z Ran 1 test in 7.725s
2022-11-23T02:58:47.0215351Z 
2022-11-23T02:58:47.0215479Z OK
2022-11-23T02:58:47.0215499Z 
2022-11-23T02:58:47.0215658Z Generating XML reports...
2022-11-23T02:58:47.0216229Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024716.xml
2022-11-23T02:58:47.0216688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0216912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0217334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0217569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0217860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83gmnfg5
2022-11-23T02:58:47.0218169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83gmnfg5/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0218188Z 
2022-11-23T02:58:47.0218335Z Running tests...
2022-11-23T02:58:47.0218631Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0219011Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0219352Z   test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0219613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92531
2022-11-23T02:58:47.0219862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92532
2022-11-23T02:58:47.0220119Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92533
2022-11-23T02:58:47.0220367Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92534
2022-11-23T02:58:47.0220779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0220993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0221396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0221628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0222032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0222244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0222653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0222934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0223335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0223544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0223953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0224163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0224569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0224780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0225185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0225412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0225702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_9cn8et6
2022-11-23T02:58:47.0226007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_9cn8et6/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0226300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0cmwrpy
2022-11-23T02:58:47.0226590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0cmwrpy/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0226927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9v9769bc
2022-11-23T02:58:47.0227233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9v9769bc/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0227523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzkc2_4d
2022-11-23T02:58:47.0227826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzkc2_4d/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0228096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0228358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0228622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0228885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0229056Z fi_getinfo: -61
2022-11-23T02:58:47.0229235Z fi_getinfo: -61
2022-11-23T02:58:47.0229413Z fi_getinfo: -61
2022-11-23T02:58:47.0229587Z fi_getinfo: -61
2022-11-23T02:58:47.0229723Z ok (7.705s)
2022-11-23T02:58:47.0229742Z 
2022-11-23T02:58:47.0230043Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0230192Z Ran 1 test in 7.706s
2022-11-23T02:58:47.0230211Z 
2022-11-23T02:58:47.0230327Z OK
2022-11-23T02:58:47.0230345Z 
2022-11-23T02:58:47.0230509Z Generating XML reports...
2022-11-23T02:58:47.0231103Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024726.xml
2022-11-23T02:58:47.0231514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0231724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0232145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0232377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0232672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps7t0x7r1
2022-11-23T02:58:47.0233020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps7t0x7r1/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0233088Z 
2022-11-23T02:58:47.0235045Z Running tests...
2022-11-23T02:58:47.0235355Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0235753Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0236094Z   test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0236351Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92890
2022-11-23T02:58:47.0236602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92891
2022-11-23T02:58:47.0236860Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92892
2022-11-23T02:58:47.0237108Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92893
2022-11-23T02:58:47.0237501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0237717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0238126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0238341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0238758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0238987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0239494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0239728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0240135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0240331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0240747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0240975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0241380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0241592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0241998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0242226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0242520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcb_46h1
2022-11-23T02:58:47.0242829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcb_46h1/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0243111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcmg7y30
2022-11-23T02:58:47.0243418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcmg7y30/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0243709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelr2bid1
2022-11-23T02:58:47.0244014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelr2bid1/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0244298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg9c2yukj
2022-11-23T02:58:47.0244599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg9c2yukj/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0244867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0245130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0245427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0245913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0246114Z fi_getinfo: -61
2022-11-23T02:58:47.0246291Z fi_getinfo: -61
2022-11-23T02:58:47.0246468Z fi_getinfo: -61
2022-11-23T02:58:47.0246646Z fi_getinfo: -61
2022-11-23T02:58:47.0246786Z ok (7.760s)
2022-11-23T02:58:47.0246806Z 
2022-11-23T02:58:47.0247088Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0247243Z Ran 1 test in 7.761s
2022-11-23T02:58:47.0247262Z 
2022-11-23T02:58:47.0247400Z OK
2022-11-23T02:58:47.0247420Z 
2022-11-23T02:58:47.0247584Z Generating XML reports...
2022-11-23T02:58:47.0248176Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024736.xml
2022-11-23T02:58:47.0248587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0248807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0249229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0249460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0249737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7up8e61m
2022-11-23T02:58:47.0250048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7up8e61m/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0250143Z 
2022-11-23T02:58:47.0250300Z Running tests...
2022-11-23T02:58:47.0250603Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0251004Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0251347Z   test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0251612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93249
2022-11-23T02:58:47.0251871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93250
2022-11-23T02:58:47.0252105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93251
2022-11-23T02:58:47.0252360Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93252
2022-11-23T02:58:47.0252779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0252998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0253416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0253646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0254060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0254273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0254684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0254896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0255296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0255514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0255926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0256152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0256632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0256846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0257257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0257485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0257766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8o94p3m
2022-11-23T02:58:47.0258080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8o94p3m/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0258375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp70ekrgy0
2022-11-23T02:58:47.0258681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp70ekrgy0/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0258972Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvo4apg7
2022-11-23T02:58:47.0259284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvo4apg7/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0259578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2jct9pc
2022-11-23T02:58:47.0259884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2jct9pc/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0260134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0260400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0260714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0260989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0261175Z fi_getinfo: -61
2022-11-23T02:58:47.0261352Z fi_getinfo: -61
2022-11-23T02:58:47.0261531Z fi_getinfo: -61
2022-11-23T02:58:47.0261688Z fi_getinfo: -61
2022-11-23T02:58:47.0261833Z ok (7.819s)
2022-11-23T02:58:47.0261853Z 
2022-11-23T02:58:47.0262157Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0262307Z Ran 1 test in 7.820s
2022-11-23T02:58:47.0262326Z 
2022-11-23T02:58:47.0262458Z OK
2022-11-23T02:58:47.0262477Z 
2022-11-23T02:58:47.0262642Z Generating XML reports...
2022-11-23T02:58:47.0263235Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024747.xml
2022-11-23T02:58:47.0263651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0263870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0264271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0264508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0264800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpoiut5w
2022-11-23T02:58:47.0265111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpoiut5w/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0265131Z 
2022-11-23T02:58:47.0265277Z Running tests...
2022-11-23T02:58:47.0265580Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0265979Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0266324Z   test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0266565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93608
2022-11-23T02:58:47.0266824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93609
2022-11-23T02:58:47.0267150Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93610
2022-11-23T02:58:47.0267401Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93611
2022-11-23T02:58:47.0267819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0268036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0268456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0268689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0269094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0269290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0269705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0269941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0270344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0270556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0270964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0271190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0271633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0271837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0272258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0272492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0272789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1tuayedb
2022-11-23T02:58:47.0273099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1tuayedb/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0273391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4olmq614
2022-11-23T02:58:47.0273697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4olmq614/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0273992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pb6kpe6
2022-11-23T02:58:47.0274300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pb6kpe6/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0274571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzw8zf8o
2022-11-23T02:58:47.0274879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzw8zf8o/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0275150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0275419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0275684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0275947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0276135Z fi_getinfo: -61
2022-11-23T02:58:47.0276316Z fi_getinfo: -61
2022-11-23T02:58:47.0276478Z fi_getinfo: -61
2022-11-23T02:58:47.0276657Z fi_getinfo: -61
2022-11-23T02:58:47.0276798Z ok (7.761s)
2022-11-23T02:58:47.0276817Z 
2022-11-23T02:58:47.0277118Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0277264Z Ran 1 test in 7.761s
2022-11-23T02:58:47.0277283Z 
2022-11-23T02:58:47.0277468Z OK
2022-11-23T02:58:47.0277487Z 
2022-11-23T02:58:47.0277650Z Generating XML reports...
2022-11-23T02:58:47.0278243Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024757.xml
2022-11-23T02:58:47.0278635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0278852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0279272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0279507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0279800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfvicysp
2022-11-23T02:58:47.0280114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfvicysp/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0280138Z 
2022-11-23T02:58:47.0280288Z Running tests...
2022-11-23T02:58:47.0280590Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0280971Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0281322Z   test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0281582Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93967
2022-11-23T02:58:47.0281840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93968
2022-11-23T02:58:47.0282144Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93969
2022-11-23T02:58:47.0282400Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93970
2022-11-23T02:58:47.0282814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0283034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0283455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0283669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0284075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0284289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0284707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0284938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0285339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0285561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0286196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0286404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0286813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0287023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0287437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0287664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0287958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzp4ex03x
2022-11-23T02:58:47.0288269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzp4ex03x/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0288650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9q_7ybw
2022-11-23T02:58:47.0288954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9q_7ybw/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0289230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0rcqpef
2022-11-23T02:58:47.0289540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0rcqpef/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0289829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphumcd33w
2022-11-23T02:58:47.0290143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphumcd33w/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0290413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0290681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0290954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0291220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0291393Z fi_getinfo: -61
2022-11-23T02:58:47.0291573Z fi_getinfo: -61
2022-11-23T02:58:47.0291746Z fi_getinfo: -61
2022-11-23T02:58:47.0291921Z fi_getinfo: -61
2022-11-23T02:58:47.0292063Z ok (7.602s)
2022-11-23T02:58:47.0292083Z 
2022-11-23T02:58:47.0292388Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0292541Z Ran 1 test in 7.603s
2022-11-23T02:58:47.0292560Z 
2022-11-23T02:58:47.0292751Z OK
2022-11-23T02:58:47.0292772Z 
2022-11-23T02:58:47.0292923Z Generating XML reports...
2022-11-23T02:58:47.0293522Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024808.xml
2022-11-23T02:58:47.0293936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0294154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0294572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0294802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0295094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82ctmoze
2022-11-23T02:58:47.0295406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82ctmoze/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0295426Z 
2022-11-23T02:58:47.0295580Z Running tests...
2022-11-23T02:58:47.0295863Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0296263Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0296617Z   test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0296875Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94318
2022-11-23T02:58:47.0297131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94319
2022-11-23T02:58:47.0297386Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94320
2022-11-23T02:58:47.0297636Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94321
2022-11-23T02:58:47.0298055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0298253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0298674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0298903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0299360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0299573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0299987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0300217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0300616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0300813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0301221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0301451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0301864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0302076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0302486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0302712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0303010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9vmpt5xn
2022-11-23T02:58:47.0303364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9vmpt5xn/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0303644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8cekiqk
2022-11-23T02:58:47.0303949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8cekiqk/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0304241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznurv5xh
2022-11-23T02:58:47.0304551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznurv5xh/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0304840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj13l302q
2022-11-23T02:58:47.0305141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj13l302q/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0305410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0305680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0305927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0306192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0306382Z fi_getinfo: -61
2022-11-23T02:58:47.0306562Z fi_getinfo: -61
2022-11-23T02:58:47.0306744Z fi_getinfo: -61
2022-11-23T02:58:47.0306920Z fi_getinfo: -61
2022-11-23T02:58:47.0307061Z ok (7.764s)
2022-11-23T02:58:47.0307081Z 
2022-11-23T02:58:47.0307386Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0307522Z Ran 1 test in 7.764s
2022-11-23T02:58:47.0307541Z 
2022-11-23T02:58:47.0307674Z OK
2022-11-23T02:58:47.0307693Z 
2022-11-23T02:58:47.0307855Z Generating XML reports...
2022-11-23T02:58:47.0308446Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024818.xml
2022-11-23T02:58:47.0308857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0309075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0309495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0309778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0310052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6y_lpro1
2022-11-23T02:58:47.0310359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6y_lpro1/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0310378Z 
2022-11-23T02:58:47.0310529Z Running tests...
2022-11-23T02:58:47.0310835Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0311236Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0311588Z   test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0311850Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94669
2022-11-23T02:58:47.0312108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94670
2022-11-23T02:58:47.0312366Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94671
2022-11-23T02:58:47.0312597Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94672
2022-11-23T02:58:47.0313021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0313238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0313657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0313932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0314346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0314558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0314969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0315182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0315590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0315805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0316222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0316448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0316854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0317072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0317486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0317716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0317994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68feaor6
2022-11-23T02:58:47.0318306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68feaor6/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0318598Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67j02wsc
2022-11-23T02:58:47.0318906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67j02wsc/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0319201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdeut51o
2022-11-23T02:58:47.0319506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdeut51o/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0319796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkcppxxs1
2022-11-23T02:58:47.0320162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkcppxxs1/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0320412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0320680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0320947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0321213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0321401Z fi_getinfo: -61
2022-11-23T02:58:47.0321582Z fi_getinfo: -61
2022-11-23T02:58:47.0321760Z fi_getinfo: -61
2022-11-23T02:58:47.0321937Z fi_getinfo: -61
2022-11-23T02:58:47.0322060Z ok (7.776s)
2022-11-23T02:58:47.0322079Z 
2022-11-23T02:58:47.0322388Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0322541Z Ran 1 test in 7.777s
2022-11-23T02:58:47.0322564Z 
2022-11-23T02:58:47.0322695Z OK
2022-11-23T02:58:47.0322714Z 
2022-11-23T02:58:47.0322877Z Generating XML reports...
2022-11-23T02:58:47.0323469Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024829.xml
2022-11-23T02:58:47.0323880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0324100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0324550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0324788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0325086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqa4jzq6
2022-11-23T02:58:47.0325391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqa4jzq6/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0325415Z 
2022-11-23T02:58:47.0325565Z Running tests...
2022-11-23T02:58:47.0326083Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0326493Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0326844Z   test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0327104Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95020
2022-11-23T02:58:47.0327347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95021
2022-11-23T02:58:47.0327605Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95022
2022-11-23T02:58:47.0327858Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95023
2022-11-23T02:58:47.0328274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0328495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0328912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0329145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0329552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0329745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0330158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0330388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0330795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0331085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0331496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0331721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0332128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0332340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0332735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0333002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0333301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_bfatna
2022-11-23T02:58:47.0333612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_bfatna/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0333910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkd_mxh68
2022-11-23T02:58:47.0334215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkd_mxh68/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0334505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wz9v47t
2022-11-23T02:58:47.0334808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wz9v47t/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0335140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx558mw0j
2022-11-23T02:58:47.0335452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx558mw0j/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0335720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0335988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0336259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0336522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0336709Z fi_getinfo: -61
2022-11-23T02:58:47.0336887Z fi_getinfo: -61
2022-11-23T02:58:47.0337046Z fi_getinfo: -61
2022-11-23T02:58:47.0337223Z fi_getinfo: -61
2022-11-23T02:58:47.0337361Z ok (7.655s)
2022-11-23T02:58:47.0337380Z 
2022-11-23T02:58:47.0337683Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0337837Z Ran 1 test in 7.655s
2022-11-23T02:58:47.0337860Z 
2022-11-23T02:58:47.0337994Z OK
2022-11-23T02:58:47.0338013Z 
2022-11-23T02:58:47.0338176Z Generating XML reports...
2022-11-23T02:58:47.0338767Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024839.xml
2022-11-23T02:58:47.0339168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0339388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0339810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0340039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0340333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphyx7o89
2022-11-23T02:58:47.0340647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphyx7o89/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0340667Z 
2022-11-23T02:58:47.0340817Z Running tests...
2022-11-23T02:58:47.0341120Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0341520Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0341921Z   test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0342180Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95371
2022-11-23T02:58:47.0342439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95372
2022-11-23T02:58:47.0342693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95373
2022-11-23T02:58:47.0342943Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95374
2022-11-23T02:58:47.0343366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0343584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0344001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0344218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0344628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0344839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0345252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0345483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0345931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0346149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0346564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0346789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0347184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0347397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0347807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0348033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0348328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpod6jjuau
2022-11-23T02:58:47.0348640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpod6jjuau/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0348935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0xwkxi6
2022-11-23T02:58:47.0349244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0xwkxi6/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0349523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp84h1msqi
2022-11-23T02:58:47.0349831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp84h1msqi/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0350122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_0zc20e
2022-11-23T02:58:47.0350424Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_0zc20e/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0350692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0350965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0351233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0351499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0351686Z fi_getinfo: -61
2022-11-23T02:58:47.0351901Z fi_getinfo: -61
2022-11-23T02:58:47.0352076Z fi_getinfo: -61
2022-11-23T02:58:47.0352251Z fi_getinfo: -61
2022-11-23T02:58:47.0352390Z ok (7.756s)
2022-11-23T02:58:47.0352409Z 
2022-11-23T02:58:47.0352714Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0352866Z Ran 1 test in 7.756s
2022-11-23T02:58:47.0352885Z 
2022-11-23T02:58:47.0353018Z OK
2022-11-23T02:58:47.0353037Z 
2022-11-23T02:58:47.0353181Z Generating XML reports...
2022-11-23T02:58:47.0353770Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024850.xml
2022-11-23T02:58:47.0354188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0354404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0354829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0355065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0355358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp32lis5o_
2022-11-23T02:58:47.0355665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp32lis5o_/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0355685Z 
2022-11-23T02:58:47.0355836Z Running tests...
2022-11-23T02:58:47.0356120Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0356563Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0356921Z   test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0357179Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95722
2022-11-23T02:58:47.0357437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95723
2022-11-23T02:58:47.0357695Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95724
2022-11-23T02:58:47.0357945Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95725
2022-11-23T02:58:47.0358360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0358558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0358979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0359213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0359624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0359838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0360257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0360486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0360886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0361101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0361492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0361723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0362130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0362343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0362750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0363030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0363325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4uicnczb
2022-11-23T02:58:47.0363638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4uicnczb/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0363914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw5jpcglt
2022-11-23T02:58:47.0364225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw5jpcglt/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0364521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrw8zcbd
2022-11-23T02:58:47.0364828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrw8zcbd/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0365117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpymsfl25p
2022-11-23T02:58:47.0365423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpymsfl25p/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0365906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0366181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0366450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0366697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0366888Z fi_getinfo: -61
2022-11-23T02:58:47.0367139Z fi_getinfo: -61
2022-11-23T02:58:47.0367325Z fi_getinfo: -61
2022-11-23T02:58:47.0367500Z fi_getinfo: -61
2022-11-23T02:58:47.0367640Z ok (7.723s)
2022-11-23T02:58:47.0367660Z 
2022-11-23T02:58:47.0367966Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0368104Z Ran 1 test in 7.723s
2022-11-23T02:58:47.0368123Z 
2022-11-23T02:58:47.0368251Z OK
2022-11-23T02:58:47.0368271Z 
2022-11-23T02:58:47.0368434Z Generating XML reports...
2022-11-23T02:58:47.0369028Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024900.xml
2022-11-23T02:58:47.0369439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0369655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0370078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0370310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0370604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpirwmgom2
2022-11-23T02:58:47.0370894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpirwmgom2/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0370917Z 
2022-11-23T02:58:47.0371066Z Running tests...
2022-11-23T02:58:47.0371369Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0371770Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0372117Z   test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0372376Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96073
2022-11-23T02:58:47.0372636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96074
2022-11-23T02:58:47.0372895Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96075
2022-11-23T02:58:47.0373126Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96076
2022-11-23T02:58:47.0373542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0373828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0374252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0374482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0374887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0375102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0375518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0375749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0376132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0376352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0376762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0376989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0377392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0377605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0378061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0378295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0378572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64dn0rqd
2022-11-23T02:58:47.0378888Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64dn0rqd/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0379184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptw89mzz6
2022-11-23T02:58:47.0379487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptw89mzz6/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0379777Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqe5q31k
2022-11-23T02:58:47.0380082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqe5q31k/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0380376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31xipydn
2022-11-23T02:58:47.0380681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31xipydn/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0380949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0381198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0381470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0381738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0381925Z fi_getinfo: -61
2022-11-23T02:58:47.0382104Z fi_getinfo: -61
2022-11-23T02:58:47.0382279Z fi_getinfo: -61
2022-11-23T02:58:47.0382456Z fi_getinfo: -61
2022-11-23T02:58:47.0382579Z ok (7.748s)
2022-11-23T02:58:47.0382599Z 
2022-11-23T02:58:47.0382904Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0383061Z Ran 1 test in 7.749s
2022-11-23T02:58:47.0383080Z 
2022-11-23T02:58:47.0383211Z OK
2022-11-23T02:58:47.0383230Z 
2022-11-23T02:58:47.0383394Z Generating XML reports...
2022-11-23T02:58:47.0383987Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024910.xml
2022-11-23T02:58:47.0384452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0384666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0385089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0385300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0385600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3qmx004
2022-11-23T02:58:47.0385912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3qmx004/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0385932Z 
2022-11-23T02:58:47.0386080Z Running tests...
2022-11-23T02:58:47.0386384Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0386783Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0387138Z   test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0387399Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96424
2022-11-23T02:58:47.0387637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96425
2022-11-23T02:58:47.0387893Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96426
2022-11-23T02:58:47.0388144Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96427
2022-11-23T02:58:47.0388599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0388822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0389242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0389477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0389883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0390096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0390489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0390718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0391121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0391335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0391743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0391972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0392386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0392598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0392988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0393217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0393512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hunl6dh
2022-11-23T02:58:47.0393827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hunl6dh/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0394121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hzpmarr
2022-11-23T02:58:47.0394430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hzpmarr/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0394778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3wvov8q
2022-11-23T02:58:47.0395082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3wvov8q/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0395375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgs9ieq2l
2022-11-23T02:58:47.0395663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgs9ieq2l/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0395932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0396203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0396470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0396731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0396924Z fi_getinfo: -61
2022-11-23T02:58:47.0397101Z fi_getinfo: -61
2022-11-23T02:58:47.0397277Z fi_getinfo: -61
2022-11-23T02:58:47.0397435Z fi_getinfo: -61
2022-11-23T02:58:47.0397574Z ok (7.765s)
2022-11-23T02:58:47.0397593Z 
2022-11-23T02:58:47.0397896Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0398048Z Ran 1 test in 7.765s
2022-11-23T02:58:47.0398067Z 
2022-11-23T02:58:47.0398198Z OK
2022-11-23T02:58:47.0398217Z 
2022-11-23T02:58:47.0398380Z Generating XML reports...
2022-11-23T02:58:47.0399022Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024921.xml
2022-11-23T02:58:47.0399445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0399643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0400066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0400302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0400596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhkkuox1
2022-11-23T02:58:47.0400904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhkkuox1/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0400924Z 
2022-11-23T02:58:47.0401072Z Running tests...
2022-11-23T02:58:47.0401376Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0401781Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0402113Z   test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0402372Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96775
2022-11-23T02:58:47.0402632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96776
2022-11-23T02:58:47.0402886Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96777
2022-11-23T02:58:47.0403137Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96778
2022-11-23T02:58:47.0403549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0403764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0404188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0404418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0404802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0405014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0405512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0405948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0406362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0406576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0406985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0407217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0407605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0407818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0408225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0408454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0408749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6o6kz6h
2022-11-23T02:58:47.0409060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6o6kz6h/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0409357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3hywtu9
2022-11-23T02:58:47.0409739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3hywtu9/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0410045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpahx4foj4
2022-11-23T02:58:47.0410335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpahx4foj4/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0410626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6o1gn631
2022-11-23T02:58:47.0410930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6o1gn631/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0411200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0411460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0411723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0411989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0412182Z fi_getinfo: -61
2022-11-23T02:58:47.0412345Z fi_getinfo: -61
2022-11-23T02:58:47.0412523Z fi_getinfo: -61
2022-11-23T02:58:47.0412695Z fi_getinfo: -61
2022-11-23T02:58:47.0412834Z ok (6.437s)
2022-11-23T02:58:47.0412854Z 
2022-11-23T02:58:47.0413157Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0413310Z Ran 1 test in 6.437s
2022-11-23T02:58:47.0413330Z 
2022-11-23T02:58:47.0413463Z OK
2022-11-23T02:58:47.0413482Z 
2022-11-23T02:58:47.0413628Z Generating XML reports...
2022-11-23T02:58:47.0414224Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024931.xml
2022-11-23T02:58:47.0414636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0414854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0415277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0415506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0415799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fb6sjuq
2022-11-23T02:58:47.0416184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fb6sjuq/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0416204Z 
2022-11-23T02:58:47.0416354Z Running tests...
2022-11-23T02:58:47.0416638Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0417035Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0417405Z   test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0417667Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97122
2022-11-23T02:58:47.0417928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97123
2022-11-23T02:58:47.0418183Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97124
2022-11-23T02:58:47.0418433Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97125
2022-11-23T02:58:47.0418852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0419067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0419469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0419701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0420110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0420366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0420790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0421020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0421423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0421641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0422034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0422265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0422675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0422887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0423301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0423527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0423823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxuhsqe1
2022-11-23T02:58:47.0424137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxuhsqe1/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0424433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6yn2tpqs
2022-11-23T02:58:47.0424727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6yn2tpqs/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0425020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hth5ebz
2022-11-23T02:58:47.0425326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hth5ebz/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0425622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4pnwet8
2022-11-23T02:58:47.0425926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4pnwet8/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0426196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0426514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0426780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0427028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0427219Z fi_getinfo: -61
2022-11-23T02:58:47.0427395Z fi_getinfo: -61
2022-11-23T02:58:47.0427571Z fi_getinfo: -61
2022-11-23T02:58:47.0427747Z fi_getinfo: -61
2022-11-23T02:58:47.0427887Z ok (7.838s)
2022-11-23T02:58:47.0427906Z 
2022-11-23T02:58:47.0428214Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0428348Z Ran 1 test in 7.838s
2022-11-23T02:58:47.0428387Z 
2022-11-23T02:58:47.0428497Z OK
2022-11-23T02:58:47.0428515Z 
2022-11-23T02:58:47.0428678Z Generating XML reports...
2022-11-23T02:58:47.0429266Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024941.xml
2022-11-23T02:58:47.0429685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0429901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0430317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0430549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0430843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftezz6ar
2022-11-23T02:58:47.0431179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftezz6ar/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0431220Z 
2022-11-23T02:58:47.0431352Z Running tests...
2022-11-23T02:58:47.0431659Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0432058Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0432416Z   test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0432675Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97481
2022-11-23T02:58:47.0432966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97482
2022-11-23T02:58:47.0433224Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97483
2022-11-23T02:58:47.0433475Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97484
2022-11-23T02:58:47.0433875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0434094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0434513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0434748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0435152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0435363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0435781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0436010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0436395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0436612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0437022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0437293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0437691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0437905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0438326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0438556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0438859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmr42ww_m
2022-11-23T02:58:47.0439152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmr42ww_m/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0439448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2jpe0_r
2022-11-23T02:58:47.0439759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2jpe0_r/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0440055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_69aqv5
2022-11-23T02:58:47.0440356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_69aqv5/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0440644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpei91ydeb
2022-11-23T02:58:47.0440952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpei91ydeb/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0441266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0441524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0441790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0442056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0442247Z fi_getinfo: -61
2022-11-23T02:58:47.0442426Z fi_getinfo: -61
2022-11-23T02:58:47.0442601Z fi_getinfo: -61
2022-11-23T02:58:47.0442777Z fi_getinfo: -61
2022-11-23T02:58:47.0442898Z ok (6.512s)
2022-11-23T02:58:47.0442940Z 
2022-11-23T02:58:47.0443224Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0443376Z Ran 1 test in 6.513s
2022-11-23T02:58:47.0443396Z 
2022-11-23T02:58:47.0443527Z OK
2022-11-23T02:58:47.0443546Z 
2022-11-23T02:58:47.0443709Z Generating XML reports...
2022-11-23T02:58:47.0444302Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024951.xml
2022-11-23T02:58:47.0444715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0444932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0445358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0445570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0446070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc4lins6i
2022-11-23T02:58:47.0446385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc4lins6i/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0446407Z 
2022-11-23T02:58:47.0446557Z Running tests...
2022-11-23T02:58:47.0446862Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0447265Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0447624Z   test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0447886Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97832
2022-11-23T02:58:47.0448226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97833
2022-11-23T02:58:47.0448463Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97834
2022-11-23T02:58:47.0448716Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97835
2022-11-23T02:58:47.0449132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0449347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0449772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0450004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0450410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0450624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0451022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0451251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0451651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0451862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0452326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0452562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0452973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0453185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0453600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0453809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0454107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8nceoug
2022-11-23T02:58:47.0454419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8nceoug/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0454711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1czknbgw
2022-11-23T02:58:47.0455026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1czknbgw/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0455320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcc8784k
2022-11-23T02:58:47.0455621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcc8784k/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0455916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgm03i8ln
2022-11-23T02:58:47.0456202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgm03i8ln/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0456472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0456741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0457007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0457274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0457459Z fi_getinfo: -61
2022-11-23T02:58:47.0457641Z fi_getinfo: -61
2022-11-23T02:58:47.0457817Z fi_getinfo: -61
2022-11-23T02:58:47.0457973Z fi_getinfo: -61
2022-11-23T02:58:47.0458116Z ok (6.569s)
2022-11-23T02:58:47.0458135Z 
2022-11-23T02:58:47.0458438Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0458641Z Ran 1 test in 6.569s
2022-11-23T02:58:47.0458661Z 
2022-11-23T02:58:47.0458791Z OK
2022-11-23T02:58:47.0458810Z 
2022-11-23T02:58:47.0458974Z Generating XML reports...
2022-11-23T02:58:47.0459567Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025000.xml
2022-11-23T02:58:47.0459976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0460172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0460597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0460828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0461123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxuj3h2rb
2022-11-23T02:58:47.0461436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxuj3h2rb/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0461456Z 
2022-11-23T02:58:47.0461604Z Running tests...
2022-11-23T02:58:47.0461907Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0462305Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0462641Z   test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0462926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98183
2022-11-23T02:58:47.0463193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98184
2022-11-23T02:58:47.0463447Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98185
2022-11-23T02:58:47.0463698Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98186
2022-11-23T02:58:47.0464115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0464331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0464751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0464983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0465371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0465590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0466000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0466232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0466634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0466849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0467258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0467484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0467890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0468087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0468498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0468725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0469019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9jdpbq0t
2022-11-23T02:58:47.0469380Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9jdpbq0t/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0469676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5t6z2jk
2022-11-23T02:58:47.0469984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5t6z2jk/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0470279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl510xjw5
2022-11-23T02:58:47.0470565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl510xjw5/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0470859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperjjo0l2
2022-11-23T02:58:47.0471162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperjjo0l2/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0471432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0471702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0471968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0472232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0472421Z fi_getinfo: -61
2022-11-23T02:58:47.0472582Z fi_getinfo: -61
2022-11-23T02:58:47.0472761Z fi_getinfo: -61
2022-11-23T02:58:47.0472935Z fi_getinfo: -61
2022-11-23T02:58:47.0473077Z ok (7.979s)
2022-11-23T02:58:47.0473096Z 
2022-11-23T02:58:47.0473440Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0473598Z Ran 1 test in 7.979s
2022-11-23T02:58:47.0473617Z 
2022-11-23T02:58:47.0473747Z OK
2022-11-23T02:58:47.0473766Z 
2022-11-23T02:58:47.0473929Z Generating XML reports...
2022-11-23T02:58:47.0474502Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025010.xml
2022-11-23T02:58:47.0474920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0475139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0475562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0475794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0476093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6jt9az3
2022-11-23T02:58:47.0476401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6jt9az3/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0476420Z 
2022-11-23T02:58:47.0476569Z Running tests...
2022-11-23T02:58:47.0476854Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0477258Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0477602Z   test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0477860Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98542
2022-11-23T02:58:47.0478115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98543
2022-11-23T02:58:47.0478369Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98544
2022-11-23T02:58:47.0478624Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98545
2022-11-23T02:58:47.0479039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0479255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0479655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0479939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0480346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0480559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0480974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0481202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0481613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0481828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0482231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0482463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0482873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0483083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0483493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0483721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0484060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspdvz43m
2022-11-23T02:58:47.0484376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspdvz43m/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0484670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk56dlcui
2022-11-23T02:58:47.0484956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk56dlcui/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0485254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2f5cefga
2022-11-23T02:58:47.0485562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2f5cefga/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0486062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpms1yjar0
2022-11-23T02:58:47.0486375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpms1yjar0/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0486651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0486919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0487189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0487456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0487635Z fi_getinfo: -61
2022-11-23T02:58:47.0487813Z fi_getinfo: -61
2022-11-23T02:58:47.0487988Z fi_getinfo: -61
2022-11-23T02:58:47.0488164Z fi_getinfo: -61
2022-11-23T02:58:47.0488305Z ok (7.742s)
2022-11-23T02:58:47.0488324Z 
2022-11-23T02:58:47.0488630Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0488788Z Ran 1 test in 7.743s
2022-11-23T02:58:47.0488808Z 
2022-11-23T02:58:47.0488920Z OK
2022-11-23T02:58:47.0488938Z 
2022-11-23T02:58:47.0489103Z Generating XML reports...
2022-11-23T02:58:47.0489701Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025020.xml
2022-11-23T02:58:47.0490113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0490328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0490842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0491075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0491369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvp0ht_qw
2022-11-23T02:58:47.0491659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvp0ht_qw/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0491701Z 
2022-11-23T02:58:47.0491832Z Running tests...
2022-11-23T02:58:47.0492137Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0492545Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0492910Z   test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0493168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98901
2022-11-23T02:58:47.0493431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98902
2022-11-23T02:58:47.0493686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98903
2022-11-23T02:58:47.0493936Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98904
2022-11-23T02:58:47.0494330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0494545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0495031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0495272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0495681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0495897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0496316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0496547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0496933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0497148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0497558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0497785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0498194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0498406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0498818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0499046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0499390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rphrsfy
2022-11-23T02:58:47.0499687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rphrsfy/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0499985Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vnw9yrr
2022-11-23T02:58:47.0500299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vnw9yrr/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0500594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqbghofz
2022-11-23T02:58:47.0500903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqbghofz/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0501251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8g3b2ivb
2022-11-23T02:58:47.0501556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8g3b2ivb/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0501823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0502090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0502335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0502607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0502794Z fi_getinfo: -61
2022-11-23T02:58:47.0502969Z fi_getinfo: -61
2022-11-23T02:58:47.0503143Z fi_getinfo: -61
2022-11-23T02:58:47.0503320Z fi_getinfo: -61
2022-11-23T02:58:47.0503459Z ok (4.603s)
2022-11-23T02:58:47.0503479Z 
2022-11-23T02:58:47.0503769Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0503922Z Ran 1 test in 4.603s
2022-11-23T02:58:47.0503942Z 
2022-11-23T02:58:47.0504073Z OK
2022-11-23T02:58:47.0504092Z 
2022-11-23T02:58:47.0504255Z Generating XML reports...
2022-11-23T02:58:47.0504846Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025031.xml
2022-11-23T02:58:47.0505259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0505520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0505951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0506164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0506462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbyhbjbjb
2022-11-23T02:58:47.0506777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbyhbjbjb/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0506797Z 
2022-11-23T02:58:47.0506945Z Running tests...
2022-11-23T02:58:47.0507249Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0507650Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0508013Z   test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0508275Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99088
2022-11-23T02:58:47.0508532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99089
2022-11-23T02:58:47.0508769Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99090
2022-11-23T02:58:47.0509021Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99091
2022-11-23T02:58:47.0509440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0509657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0510080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0510313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0510723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0510936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0511324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0511540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0512016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0512247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0512660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0512892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0513300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0513515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0513928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0514139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0514437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_f625pcs
2022-11-23T02:58:47.0514747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_f625pcs/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0515041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0abetr7
2022-11-23T02:58:47.0515352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0abetr7/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0515645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvma4vn1e
2022-11-23T02:58:47.0515994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvma4vn1e/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0516295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw701a3xg
2022-11-23T02:58:47.0516598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw701a3xg/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0516850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0517122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0517389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0517654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0517840Z fi_getinfo: -61
2022-11-23T02:58:47.0518019Z fi_getinfo: -61
2022-11-23T02:58:47.0518195Z fi_getinfo: -61
2022-11-23T02:58:47.0518352Z fi_getinfo: -61
2022-11-23T02:58:47.0518493Z ok (4.606s)
2022-11-23T02:58:47.0518512Z 
2022-11-23T02:58:47.0518820Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0518973Z Ran 1 test in 4.606s
2022-11-23T02:58:47.0518992Z 
2022-11-23T02:58:47.0519123Z OK
2022-11-23T02:58:47.0519141Z 
2022-11-23T02:58:47.0519304Z Generating XML reports...
2022-11-23T02:58:47.0519894Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025038.xml
2022-11-23T02:58:47.0520311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0520509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0520930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0521162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0521462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphagwx45u
2022-11-23T02:58:47.0521773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphagwx45u/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0521793Z 
2022-11-23T02:58:47.0521940Z Running tests...
2022-11-23T02:58:47.0522243Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0522701Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0523057Z   test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0523294Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99275
2022-11-23T02:58:47.0523551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99276
2022-11-23T02:58:47.0523804Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99277
2022-11-23T02:58:47.0524058Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99278
2022-11-23T02:58:47.0524471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0524688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0525111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0525347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0525939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0526164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0526581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0526884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0527305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0527519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0527930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0528161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0528564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0528759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0529172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0529400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0529700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8r6yn_op
2022-11-23T02:58:47.0530010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8r6yn_op/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0530306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj22sk32m
2022-11-23T02:58:47.0530614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj22sk32m/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0530906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsryckxp4
2022-11-23T02:58:47.0531216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsryckxp4/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0531490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mg7zgpa
2022-11-23T02:58:47.0531794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mg7zgpa/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0532068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0532337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0532602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0532974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0533163Z fi_getinfo: -61
2022-11-23T02:58:47.0533346Z fi_getinfo: -61
2022-11-23T02:58:47.0533505Z fi_getinfo: -61
2022-11-23T02:58:47.0533685Z fi_getinfo: -61
2022-11-23T02:58:47.0533826Z ok (4.398s)
2022-11-23T02:58:47.0533846Z 
2022-11-23T02:58:47.0534150Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0534304Z Ran 1 test in 4.398s
2022-11-23T02:58:47.0534323Z 
2022-11-23T02:58:47.0534454Z OK
2022-11-23T02:58:47.0534474Z 
2022-11-23T02:58:47.0534636Z Generating XML reports...
2022-11-23T02:58:47.0535213Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025045.xml
2022-11-23T02:58:47.0535624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0535840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0536265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0536495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0536789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yyhpylq
2022-11-23T02:58:47.0537099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yyhpylq/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0537118Z 
2022-11-23T02:58:47.0537269Z Running tests...
2022-11-23T02:58:47.0537619Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0538010Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0538355Z   test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0538617Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99450
2022-11-23T02:58:47.0538873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99451
2022-11-23T02:58:47.0539127Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99452
2022-11-23T02:58:47.0539376Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99453
2022-11-23T02:58:47.0539786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0540001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0540405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0540638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0541043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0541259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0541672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0541902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0542304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0542515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0542939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0543149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0543552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0543818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0544234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0544461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0544758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3n_3ei20
2022-11-23T02:58:47.0545066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3n_3ei20/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0545366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptweujmd
2022-11-23T02:58:47.0545658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptweujmd/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0545950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp022jubws
2022-11-23T02:58:47.0546259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp022jubws/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0546555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6jb2okap
2022-11-23T02:58:47.0546856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6jb2okap/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0547127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0547395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0547659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0547972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0548147Z fi_getinfo: -61
2022-11-23T02:58:47.0548327Z fi_getinfo: -61
2022-11-23T02:58:47.0548503Z fi_getinfo: -61
2022-11-23T02:58:47.0548671Z fi_getinfo: -61
2022-11-23T02:58:47.0548814Z ok (4.640s)
2022-11-23T02:58:47.0548837Z 
2022-11-23T02:58:47.0549143Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0549298Z Ran 1 test in 4.640s
2022-11-23T02:58:47.0549318Z 
2022-11-23T02:58:47.0549429Z OK
2022-11-23T02:58:47.0549447Z 
2022-11-23T02:58:47.0549612Z Generating XML reports...
2022-11-23T02:58:47.0550202Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025052.xml
2022-11-23T02:58:47.0550615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0550834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0551257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0551489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0551785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0ubiiht
2022-11-23T02:58:47.0552100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0ubiiht/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0552120Z 
2022-11-23T02:58:47.0552251Z Running tests...
2022-11-23T02:58:47.0552559Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0552959Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0553306Z   test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0553568Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99637
2022-11-23T02:58:47.0553827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99638
2022-11-23T02:58:47.0554083Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99639
2022-11-23T02:58:47.0554390Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99640
2022-11-23T02:58:47.0554786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0555005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0555428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0555659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0556066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0556279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0556693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0556923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0557332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0557527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0557937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0558166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0558573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0558828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0559245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0559473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0559772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6usgtn9d
2022-11-23T02:58:47.0560062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6usgtn9d/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0560357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyw_6od6g
2022-11-23T02:58:47.0560668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyw_6od6g/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0560961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvtlfuak4
2022-11-23T02:58:47.0561271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvtlfuak4/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0561561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1r6smb0
2022-11-23T02:58:47.0561861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1r6smb0/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0562133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0562400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0562647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0562913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0563097Z fi_getinfo: -61
2022-11-23T02:58:47.0563273Z fi_getinfo: -61
2022-11-23T02:58:47.0563446Z fi_getinfo: -61
2022-11-23T02:58:47.0563621Z fi_getinfo: -61
2022-11-23T02:58:47.0563761Z ok (5.844s)
2022-11-23T02:58:47.0563785Z 
2022-11-23T02:58:47.0564072Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0564227Z Ran 1 test in 5.844s
2022-11-23T02:58:47.0564246Z 
2022-11-23T02:58:47.0564376Z OK
2022-11-23T02:58:47.0564395Z 
2022-11-23T02:58:47.0564557Z Generating XML reports...
2022-11-23T02:58:47.0565215Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025100.xml
2022-11-23T02:58:47.0565625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0566051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0566483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0566715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0566997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6k0cfrtx
2022-11-23T02:58:47.0567310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6k0cfrtx/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0567330Z 
2022-11-23T02:58:47.0567478Z Running tests...
2022-11-23T02:58:47.0567785Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0568191Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0568547Z   test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0568805Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99984
2022-11-23T02:58:47.0569062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99985
2022-11-23T02:58:47.0569298Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99986
2022-11-23T02:58:47.0569664Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99987
2022-11-23T02:58:47.0570092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0570307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0570729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0570961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0571365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0571577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0571989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0572205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0572608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0572823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0573231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0573462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0573870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0574083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0574495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0574708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0575005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2ayfqjs
2022-11-23T02:58:47.0575315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2ayfqjs/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0575607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcc4pz0q
2022-11-23T02:58:47.0575982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcc4pz0q/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0576273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzjtl_86
2022-11-23T02:58:47.0576579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzjtl_86/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0576871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1kc1_3qt
2022-11-23T02:58:47.0577170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1kc1_3qt/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0577422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0577690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0577958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0578228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0578415Z fi_getinfo: -61
2022-11-23T02:58:47.0578596Z fi_getinfo: -61
2022-11-23T02:58:47.0578774Z fi_getinfo: -61
2022-11-23T02:58:47.0578932Z fi_getinfo: -61
2022-11-23T02:58:47.0579078Z ok (5.938s)
2022-11-23T02:58:47.0579098Z 
2022-11-23T02:58:47.0579402Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0579555Z Ran 1 test in 5.938s
2022-11-23T02:58:47.0579574Z 
2022-11-23T02:58:47.0579707Z OK
2022-11-23T02:58:47.0579725Z 
2022-11-23T02:58:47.0579888Z Generating XML reports...
2022-11-23T02:58:47.0580523Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025108.xml
2022-11-23T02:58:47.0580944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0581167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0581568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0581800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0582093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4oao5gg
2022-11-23T02:58:47.0582400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4oao5gg/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0582420Z 
2022-11-23T02:58:47.0582568Z Running tests...
2022-11-23T02:58:47.0582876Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0583279Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0583647Z   test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0583896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100331
2022-11-23T02:58:47.0584156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100332
2022-11-23T02:58:47.0584412Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100333
2022-11-23T02:58:47.0584665Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100334
2022-11-23T02:58:47.0585078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0585299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0585721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0585955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0586363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0586611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0587033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0587262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0587669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0587890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0588304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0588512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0588912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0589126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0589532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0589756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0590047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_yy0n52m
2022-11-23T02:58:47.0590351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_yy0n52m/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0590687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2_shovqg
2022-11-23T02:58:47.0590999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2_shovqg/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0591271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11ye_hjs
2022-11-23T02:58:47.0591575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11ye_hjs/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0591864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dhjg12y
2022-11-23T02:58:47.0592167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dhjg12y/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0592434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0592695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0592961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0593225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0593397Z fi_getinfo: -61
2022-11-23T02:58:47.0593573Z fi_getinfo: -61
2022-11-23T02:58:47.0593742Z fi_getinfo: -61
2022-11-23T02:58:47.0593913Z fi_getinfo: -61
2022-11-23T02:58:47.0594053Z ok (5.855s)
2022-11-23T02:58:47.0594073Z 
2022-11-23T02:58:47.0594373Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0594524Z Ran 1 test in 5.855s
2022-11-23T02:58:47.0594543Z 
2022-11-23T02:58:47.0594656Z OK
2022-11-23T02:58:47.0594691Z 
2022-11-23T02:58:47.0594836Z Generating XML reports...
2022-11-23T02:58:47.0595422Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025117.xml
2022-11-23T02:58:47.0595831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0596044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0596463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0596690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0597050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9tlnyqy
2022-11-23T02:58:47.0597354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9tlnyqy/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0597374Z 
2022-11-23T02:58:47.0597505Z Running tests...
2022-11-23T02:58:47.0597805Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0598199Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0598563Z   test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0598823Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100678
2022-11-23T02:58:47.0599078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100679
2022-11-23T02:58:47.0599333Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100680
2022-11-23T02:58:47.0599588Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100681
2022-11-23T02:58:47.0599997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0600196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0600614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0600841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0601286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0601503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0601914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0602145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0602542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0602736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0603150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0603374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0603772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0603984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0604395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0604618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0604915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3tsfv49
2022-11-23T02:58:47.0605219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3tsfv49/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0605493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7cd_yfw
2022-11-23T02:58:47.0606000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7cd_yfw/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0606298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybfi2of7
2022-11-23T02:58:47.0606608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybfi2of7/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0606893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vj2zudf
2022-11-23T02:58:47.0607193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vj2zudf/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0607532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0607797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0608043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0608303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0608496Z fi_getinfo: -61
2022-11-23T02:58:47.0608670Z fi_getinfo: -61
2022-11-23T02:58:47.0608840Z fi_getinfo: -61
2022-11-23T02:58:47.0609016Z fi_getinfo: -61
2022-11-23T02:58:47.0609154Z ok (5.958s)
2022-11-23T02:58:47.0609174Z 
2022-11-23T02:58:47.0609476Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0609610Z Ran 1 test in 5.959s
2022-11-23T02:58:47.0609629Z 
2022-11-23T02:58:47.0609758Z OK
2022-11-23T02:58:47.0609777Z 
2022-11-23T02:58:47.0609941Z Generating XML reports...
2022-11-23T02:58:47.0610532Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025125.xml
2022-11-23T02:58:47.0610944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0611158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0611579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0611866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0612151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf14az_zu
2022-11-23T02:58:47.0612458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf14az_zu/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0612478Z 
2022-11-23T02:58:47.0612628Z Running tests...
2022-11-23T02:58:47.0612930Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0613325Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0613694Z   test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0613953Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101025
2022-11-23T02:58:47.0614207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101026
2022-11-23T02:58:47.0614465Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101027
2022-11-23T02:58:47.0614700Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101028
2022-11-23T02:58:47.0615112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0615329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0615746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0615972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0616369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0616579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0616992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0617206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0617606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0617819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0618282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0618506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0618910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0619118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0619520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0619749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0620027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfcu830lx
2022-11-23T02:58:47.0620336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfcu830lx/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0620631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplryi5eg2
2022-11-23T02:58:47.0620935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplryi5eg2/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0621223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp259y523x
2022-11-23T02:58:47.0621521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp259y523x/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0621805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gddjyys
2022-11-23T02:58:47.0622155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gddjyys/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0622412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0622678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0622941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0623205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0623395Z fi_getinfo: -61
2022-11-23T02:58:47.0623569Z fi_getinfo: -61
2022-11-23T02:58:47.0623741Z fi_getinfo: -61
2022-11-23T02:58:47.0623914Z fi_getinfo: -61
2022-11-23T02:58:47.0624037Z ok (5.805s)
2022-11-23T02:58:47.0624058Z 
2022-11-23T02:58:47.0624359Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0624509Z Ran 1 test in 5.806s
2022-11-23T02:58:47.0624528Z 
2022-11-23T02:58:47.0624655Z OK
2022-11-23T02:58:47.0624678Z 
2022-11-23T02:58:47.0624839Z Generating XML reports...
2022-11-23T02:58:47.0625427Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025134.xml
2022-11-23T02:58:47.0625829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0626045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0626446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0626677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0626969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpimfxe8ba
2022-11-23T02:58:47.0627270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpimfxe8ba/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0627290Z 
2022-11-23T02:58:47.0627437Z Running tests...
2022-11-23T02:58:47.0627738Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0628134Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0628496Z   test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0628809Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101372
2022-11-23T02:58:47.0629050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101373
2022-11-23T02:58:47.0629304Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101374
2022-11-23T02:58:47.0629554Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101375
2022-11-23T02:58:47.0629970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0630187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0630605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0630835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0631241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0631433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0631842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0632069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0632470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0632726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0633190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0633417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0633823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0634036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0634426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0634651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0634945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqu_dcvj
2022-11-23T02:58:47.0635256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqu_dcvj/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0635550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvd5izgaz
2022-11-23T02:58:47.0635856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvd5izgaz/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0636148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjgq5qjd
2022-11-23T02:58:47.0636457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjgq5qjd/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0636729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjp7n61z5
2022-11-23T02:58:47.0637030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjp7n61z5/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0637295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0637557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0637817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0638078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0638263Z fi_getinfo: -61
2022-11-23T02:58:47.0638437Z fi_getinfo: -61
2022-11-23T02:58:47.0638668Z fi_getinfo: -61
2022-11-23T02:58:47.0638844Z fi_getinfo: -61
2022-11-23T02:58:47.0638979Z ok (5.789s)
2022-11-23T02:58:47.0638998Z 
2022-11-23T02:58:47.0639306Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0639453Z Ran 1 test in 5.789s
2022-11-23T02:58:47.0639472Z 
2022-11-23T02:58:47.0639598Z OK
2022-11-23T02:58:47.0639617Z 
2022-11-23T02:58:47.0639776Z Generating XML reports...
2022-11-23T02:58:47.0640364Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025142.xml
2022-11-23T02:58:47.0640762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0640978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0641396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0641630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0641920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkvhhv__a
2022-11-23T02:58:47.0642224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkvhhv__a/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0642243Z 
2022-11-23T02:58:47.0642387Z Running tests...
2022-11-23T02:58:47.0642688Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0643085Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0643486Z   test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0643756Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101719
2022-11-23T02:58:47.0644012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101720
2022-11-23T02:58:47.0644269Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101721
2022-11-23T02:58:47.0644517Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101722
2022-11-23T02:58:47.0644931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0645144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0645564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0645986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0646404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0646615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0647026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0647256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0647652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0647862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0648267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0648480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0648884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0649113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0649525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0649843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0650136Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxdrj6xxu
2022-11-23T02:58:47.0650446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxdrj6xxu/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0650737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd49hon6o
2022-11-23T02:58:47.0651041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd49hon6o/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0651316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7tidwgdl
2022-11-23T02:58:47.0651622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7tidwgdl/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0651909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptdfpq3yx
2022-11-23T02:58:47.0652217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptdfpq3yx/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0652486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0652750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0653013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0653273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0653462Z fi_getinfo: -61
2022-11-23T02:58:47.0653623Z fi_getinfo: -61
2022-11-23T02:58:47.0653854Z fi_getinfo: -61
2022-11-23T02:58:47.0654029Z fi_getinfo: -61
2022-11-23T02:58:47.0654167Z ok (5.902s)
2022-11-23T02:58:47.0654187Z 
2022-11-23T02:58:47.0654488Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0654638Z Ran 1 test in 5.902s
2022-11-23T02:58:47.0654662Z 
2022-11-23T02:58:47.0654791Z OK
2022-11-23T02:58:47.0654810Z 
2022-11-23T02:58:47.0654956Z Generating XML reports...
2022-11-23T02:58:47.0655548Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025151.xml
2022-11-23T02:58:47.0655959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0656171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0656596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0656826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0657116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9guea9bz
2022-11-23T02:58:47.0657422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9guea9bz/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0657446Z 
2022-11-23T02:58:47.0657593Z Running tests...
2022-11-23T02:58:47.0657881Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0658275Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0658617Z   test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0658875Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102066
2022-11-23T02:58:47.0659132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102067
2022-11-23T02:58:47.0659387Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102068
2022-11-23T02:58:47.0659638Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102069
2022-11-23T02:58:47.0660053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0660302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0660724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0660952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0661351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0661560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0661976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0662205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0662601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0662816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0663212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0663439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0663832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0664043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0664501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0664734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0665028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9rnewn5_
2022-11-23T02:58:47.0665334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9rnewn5_/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0665612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpts4nb98t
2022-11-23T02:58:47.0665919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpts4nb98t/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0666210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3z4mgxh
2022-11-23T02:58:47.0666511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3z4mgxh/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0666798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg16q67rj
2022-11-23T02:58:47.0667098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg16q67rj/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0667363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0667628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0667895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0668142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0668329Z fi_getinfo: -61
2022-11-23T02:58:47.0668503Z fi_getinfo: -61
2022-11-23T02:58:47.0668674Z fi_getinfo: -61
2022-11-23T02:58:47.0668847Z fi_getinfo: -61
2022-11-23T02:58:47.0668984Z ok (7.661s)
2022-11-23T02:58:47.0669004Z 
2022-11-23T02:58:47.0669307Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0669440Z Ran 1 test in 7.662s
2022-11-23T02:58:47.0669463Z 
2022-11-23T02:58:47.0669594Z OK
2022-11-23T02:58:47.0669613Z 
2022-11-23T02:58:47.0669776Z Generating XML reports...
2022-11-23T02:58:47.0670365Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025159.xml
2022-11-23T02:58:47.0670777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0671046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0671468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0671696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0671986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkr4blmrz
2022-11-23T02:58:47.0672282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkr4blmrz/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0672302Z 
2022-11-23T02:58:47.0672448Z Running tests...
2022-11-23T02:58:47.0672751Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0673146Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0673497Z   test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0673758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102425
2022-11-23T02:58:47.0674015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102426
2022-11-23T02:58:47.0674268Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102427
2022-11-23T02:58:47.0674500Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102428
2022-11-23T02:58:47.0674960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0675186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0675604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0675833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0676236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0676446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0676860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0677089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0677470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0677682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0678089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0678313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0678722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0678933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0679338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0679560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0679839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxpq46ck
2022-11-23T02:58:47.0680152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxpq46ck/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0680444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpufns_9
2022-11-23T02:58:47.0680755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpufns_9/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0681099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcsxpduf
2022-11-23T02:58:47.0681405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcsxpduf/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0681687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1y15fr2v
2022-11-23T02:58:47.0681988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1y15fr2v/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0682251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0682503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0682766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0683026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0683211Z fi_getinfo: -61
2022-11-23T02:58:47.0683391Z fi_getinfo: -61
2022-11-23T02:58:47.0683565Z fi_getinfo: -61
2022-11-23T02:58:47.0683741Z fi_getinfo: -61
2022-11-23T02:58:47.0683862Z ok (7.748s)
2022-11-23T02:58:47.0683897Z 
2022-11-23T02:58:47.0684181Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0684330Z Ran 1 test in 7.748s
2022-11-23T02:58:47.0684349Z 
2022-11-23T02:58:47.0684479Z OK
2022-11-23T02:58:47.0684498Z 
2022-11-23T02:58:47.0684659Z Generating XML reports...
2022-11-23T02:58:47.0685248Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025210.xml
2022-11-23T02:58:47.0685927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0686165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0686595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0686811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0687101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ln85mbh
2022-11-23T02:58:47.0687405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ln85mbh/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0687426Z 
2022-11-23T02:58:47.0687569Z Running tests...
2022-11-23T02:58:47.0687870Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0688264Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0688611Z   test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0688871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102776
2022-11-23T02:58:47.0689111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102777
2022-11-23T02:58:47.0689370Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102778
2022-11-23T02:58:47.0689622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102779
2022-11-23T02:58:47.0690033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0690245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0690661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0690893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0691293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0691504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0691898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0692196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0692595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0692805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0693214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0693442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0693850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0694059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0694466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0694679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0694974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5xbqwxg
2022-11-23T02:58:47.0695266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi9khw6q1
2022-11-23T02:58:47.0695570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5xbqwxg/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0695871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi9khw6q1/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0696208Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxdge2e5g
2022-11-23T02:58:47.0696522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxdge2e5g/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0696812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ctp12wd
2022-11-23T02:58:47.0697100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ctp12wd/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0697367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0697631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0697893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0698149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0698336Z fi_getinfo: -61
2022-11-23T02:58:47.0698478Z ok (4.404s)
2022-11-23T02:58:47.0698498Z 
2022-11-23T02:58:47.0698797Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0698933Z Ran 1 test in 4.405s
2022-11-23T02:58:47.0698952Z 
2022-11-23T02:58:47.0699081Z OK
2022-11-23T02:58:47.0699100Z 
2022-11-23T02:58:47.0699259Z Generating XML reports...
2022-11-23T02:58:47.0699852Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025220.xml
2022-11-23T02:58:47.0700258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0700469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0700885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0701114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0701410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvr9lrpog
2022-11-23T02:58:47.0701702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvr9lrpog/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0701722Z 
2022-11-23T02:58:47.0701867Z Running tests...
2022-11-23T02:58:47.0702225Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0702621Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0702958Z   test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0703214Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102948
2022-11-23T02:58:47.0703469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102949
2022-11-23T02:58:47.0703724Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102950
2022-11-23T02:58:47.0703958Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102951
2022-11-23T02:58:47.0704368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0704583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0705005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0705233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0705631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0705839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0706251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0706525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0706915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0707125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0707540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0707765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0708169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0708378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0708786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0709016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0709297Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwsgi5ym
2022-11-23T02:58:47.0709605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwsgi5ym/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0709896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgp8pu8l1
2022-11-23T02:58:47.0710204Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgp8pu8l1/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0710490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdy52xbmi
2022-11-23T02:58:47.0710790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdy52xbmi/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0711078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgce0mdnp
2022-11-23T02:58:47.0711381Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgce0mdnp/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0711647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0711899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0712160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0712492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0712677Z fi_getinfo: -61
2022-11-23T02:58:47.0712850Z fi_getinfo: -61
2022-11-23T02:58:47.0713022Z fi_getinfo: -61
2022-11-23T02:58:47.0713194Z fi_getinfo: -61
2022-11-23T02:58:47.0713316Z ok (7.748s)
2022-11-23T02:58:47.0713351Z 
2022-11-23T02:58:47.0713635Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0713783Z Ran 1 test in 7.748s
2022-11-23T02:58:47.0713802Z 
2022-11-23T02:58:47.0713929Z OK
2022-11-23T02:58:47.0713949Z 
2022-11-23T02:58:47.0714115Z Generating XML reports...
2022-11-23T02:58:47.0714703Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025227.xml
2022-11-23T02:58:47.0715109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0715325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0715744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0715957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0716252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphizddjkk
2022-11-23T02:58:47.0716560Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphizddjkk/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0716581Z 
2022-11-23T02:58:47.0716724Z Running tests...
2022-11-23T02:58:47.0717078Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0717484Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0717827Z   test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0718087Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103307
2022-11-23T02:58:47.0718329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103308
2022-11-23T02:58:47.0718582Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103309
2022-11-23T02:58:47.0718831Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103310
2022-11-23T02:58:47.0719241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0719456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0719872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0720100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0720500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0720716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0721112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0721339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0721738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0721946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0722355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0722577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0722985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0723246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0723638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0723862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0724153Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fxn0zez
2022-11-23T02:58:47.0724458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fxn0zez/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0724753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0igjm10w
2022-11-23T02:58:47.0725055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0igjm10w/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0725343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkgajixs
2022-11-23T02:58:47.0725651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkgajixs/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0726148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp060p461e
2022-11-23T02:58:47.0726433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp060p461e/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0726698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0726961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0727292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0727562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0727748Z skip: Need at least 4 CUDA devices (4.421s)
2022-11-23T02:58:47.0727768Z 
2022-11-23T02:58:47.0728079Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0728234Z Ran 1 test in 4.421s
2022-11-23T02:58:47.0728254Z 
2022-11-23T02:58:47.0728398Z OK (skipped=1)
2022-11-23T02:58:47.0728418Z 
2022-11-23T02:58:47.0728561Z Generating XML reports...
2022-11-23T02:58:47.0729149Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025238.xml
2022-11-23T02:58:47.0729558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0729768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0730188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0730418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0730710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqe6ccbyu
2022-11-23T02:58:47.0731021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqe6ccbyu/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0731041Z 
2022-11-23T02:58:47.0731187Z Running tests...
2022-11-23T02:58:47.0731476Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0731871Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0732222Z   test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0732483Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103478
2022-11-23T02:58:47.0732737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103479
2022-11-23T02:58:47.0733029Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103480
2022-11-23T02:58:47.0733284Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103481
2022-11-23T02:58:47.0733798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0733996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0734419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0734647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0735047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0735262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0735672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0735902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0736304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0736498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0736906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0737133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0737538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0737797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0738212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0738434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0738728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmjqbah4
2022-11-23T02:58:47.0739042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmjqbah4/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0739320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwn5pvebg
2022-11-23T02:58:47.0739625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwn5pvebg/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0739913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7kb4atww
2022-11-23T02:58:47.0740217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7kb4atww/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0740506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0p3u793h
2022-11-23T02:58:47.0740804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0p3u793h/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0741069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0741337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0741602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0741849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0742036Z skip: Need at least 4 CUDA devices (4.422s)
2022-11-23T02:58:47.0742056Z 
2022-11-23T02:58:47.0742364Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0742514Z Ran 1 test in 4.422s
2022-11-23T02:58:47.0742533Z 
2022-11-23T02:58:47.0742681Z OK (skipped=1)
2022-11-23T02:58:47.0742701Z 
2022-11-23T02:58:47.0742861Z Generating XML reports...
2022-11-23T02:58:47.0743449Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025245.xml
2022-11-23T02:58:47.0743859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0744110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0744532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0744760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0745052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx437w919
2022-11-23T02:58:47.0745358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx437w919/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0745378Z 
2022-11-23T02:58:47.0745523Z Running tests...
2022-11-23T02:58:47.0745823Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0746219Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0746577Z   test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0746819Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103649
2022-11-23T02:58:47.0747074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103650
2022-11-23T02:58:47.0747327Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103651
2022-11-23T02:58:47.0747575Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103652
2022-11-23T02:58:47.0748031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0748251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0748670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0748898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0749291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0749502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0749918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0750144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0750548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0750761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0751172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0751396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0751799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0751992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0752399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0752624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0752915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjm7rubga
2022-11-23T02:58:47.0753228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjm7rubga/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0753519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8890i7kn
2022-11-23T02:58:47.0753821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8890i7kn/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0754166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdqcqz338
2022-11-23T02:58:47.0754467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdqcqz338/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0754739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjkpfzaoq
2022-11-23T02:58:47.0755042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjkpfzaoq/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0755307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0755576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0755839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0756098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0756285Z fi_getinfo: -61
2022-11-23T02:58:47.0756465Z fi_getinfo: -61
2022-11-23T02:58:47.0756624Z fi_getinfo: -61
2022-11-23T02:58:47.0756796Z fi_getinfo: -61
2022-11-23T02:58:47.0756931Z ok (4.531s)
2022-11-23T02:58:47.0756951Z 
2022-11-23T02:58:47.0757250Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0757399Z Ran 1 test in 4.531s
2022-11-23T02:58:47.0757419Z 
2022-11-23T02:58:47.0757548Z OK
2022-11-23T02:58:47.0757567Z 
2022-11-23T02:58:47.0757727Z Generating XML reports...
2022-11-23T02:58:47.0758299Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025252.xml
2022-11-23T02:58:47.0758759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0758982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0759402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0759632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0759924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaktwr1kx
2022-11-23T02:58:47.0760230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaktwr1kx/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0760250Z 
2022-11-23T02:58:47.0760395Z Running tests...
2022-11-23T02:58:47.0760692Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0761072Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0761407Z   test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0761665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103836
2022-11-23T02:58:47.0761918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103837
2022-11-23T02:58:47.0762178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103838
2022-11-23T02:58:47.0762427Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103839
2022-11-23T02:58:47.0762838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0763050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0763449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0763684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0764085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0764295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0764702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0764982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0765383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0765592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0766218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0766428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0766836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0767043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0767460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0767690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0767984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkes40d2d
2022-11-23T02:58:47.0768288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkes40d2d/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0768578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiy0leg1w
2022-11-23T02:58:47.0768866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiy0leg1w/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0769228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrgzvxnm
2022-11-23T02:58:47.0769547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrgzvxnm/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0769834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6gmpw8j
2022-11-23T02:58:47.0770140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6gmpw8j/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0770404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0770667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0770929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0771190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0771362Z fi_getinfo: -61
2022-11-23T02:58:47.0771545Z fi_getinfo: -61
2022-11-23T02:58:47.0771717Z fi_getinfo: -61
2022-11-23T02:58:47.0771887Z fi_getinfo: -61
2022-11-23T02:58:47.0772056Z On WorkerInfo(id=1, name=worker1):
2022-11-23T02:58:47.0784265Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f3d0251b59b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f3d02516dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7f3d0e72413d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f3d0e72567f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f3d0e726ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f3d0ea1bb7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2a0cafe (0x7f3d053d2afe in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2a0cc06 (0x7f3d053d2c06 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f3d0f52dc58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x35f0350 (0x7f3d10da0350 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x35f0ac9 (0x7f3d10da0ac9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f3d0f567e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2ff562 (0x7f3d1a48e562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2ff956 (0x7f3d1a48e956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x5620a956bc68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x5620a9527499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x5620a95275fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x5620a94d34b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x5620a9570098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x5620a951d742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x5620a94d5faa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x5620a9571774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x5620a951d742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x5620a94d5faa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xaa8dba (0x7f3d1ac37dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f3d1ac35ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f3d1ac392d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f3d1ac3ab16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f3d121db54c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f3d1ac390c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4a249d3 (0x7f3d121d49d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f3d121d5568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f3d121cf867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x4a54552 (0x7f3d12204552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f3d0250990b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7f3d32119bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7f3d527746db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7f3d5249d61f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T02:58:47.0784591Z Traceback (most recent call last):
2022-11-23T02:58:47.0784996Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:58:47.0785229Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:58:47.0785673Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2022-11-23T02:58:47.0785836Z     return x.cpu() + y.cuda()
2022-11-23T02:58:47.0786124Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-11-23T02:58:47.0786493Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-11-23T02:58:47.0787102Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f3d0251b59b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:47.0787763Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f3d02516dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:47.0788303Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7f3d0e72413d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0788804Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f3d0e72567f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0789404Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f3d0e726ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0789964Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f3d0ea1bb7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0790391Z frame #6: <unknown function> + 0x2a0cafe (0x7f3d053d2afe in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:47.0790810Z frame #7: <unknown function> + 0x2a0cc06 (0x7f3d053d2c06 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:47.0791401Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f3d0f52dc58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0791821Z frame #9: <unknown function> + 0x35f0350 (0x7f3d10da0350 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0792240Z frame #10: <unknown function> + 0x35f0ac9 (0x7f3d10da0ac9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0792767Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f3d0f567e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0793249Z frame #12: <unknown function> + 0x2ff562 (0x7f3d1a48e562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0793670Z frame #13: <unknown function> + 0x2ff956 (0x7f3d1a48e956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0793906Z frame #14: <unknown function> + 0x1ddc68 (0x5620a956bc68 in /opt/conda/bin/python)
2022-11-23T02:58:47.0794115Z frame #15: <unknown function> + 0x199499 (0x5620a9527499 in /opt/conda/bin/python)
2022-11-23T02:58:47.0794339Z frame #16: <unknown function> + 0x1995fa (0x5620a95275fa in /opt/conda/bin/python)
2022-11-23T02:58:47.0794562Z frame #17: PyNumber_Add + 0x41 (0x5620a94d34b1 in /opt/conda/bin/python)
2022-11-23T02:58:47.0794803Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x5620a9570098 in /opt/conda/bin/python)
2022-11-23T02:58:47.0795032Z frame #19: <unknown function> + 0x18f742 (0x5620a951d742 in /opt/conda/bin/python)
2022-11-23T02:58:47.0795255Z frame #20: _PyObject_Call + 0x20a (0x5620a94d5faa in /opt/conda/bin/python)
2022-11-23T02:58:47.0795498Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x5620a9571774 in /opt/conda/bin/python)
2022-11-23T02:58:47.0795720Z frame #22: <unknown function> + 0x18f742 (0x5620a951d742 in /opt/conda/bin/python)
2022-11-23T02:58:47.0795923Z frame #23: _PyObject_Call + 0x20a (0x5620a94d5faa in /opt/conda/bin/python)
2022-11-23T02:58:47.0796408Z frame #24: <unknown function> + 0xaa8dba (0x7f3d1ac37dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0796981Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f3d1ac35ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0797675Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f3d1ac392d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0798407Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f3d1ac3ab16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0799260Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f3d121db54c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0800088Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f3d1ac390c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0800511Z frame #30: <unknown function> + 0x4a249d3 (0x7f3d121d49d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0801215Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f3d121d5568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0801887Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f3d121cf867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0802303Z frame #33: <unknown function> + 0x4a54552 (0x7f3d12204552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0802812Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f3d0250990b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:47.0803067Z frame #35: <unknown function> + 0xdbbf4 (0x7f3d32119bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:58:47.0803427Z frame #36: <unknown function> + 0x76db (0x7f3d527746db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:58:47.0803737Z frame #37: clone + 0x3f (0x7f3d5249d61f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:58:47.0803759Z 
2022-11-23T02:58:47.0803777Z 
2022-11-23T02:58:47.0803934Z On WorkerInfo(id=0, name=worker0):
2022-11-23T02:58:47.0816427Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fb50a1d859b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fb50a1d3dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7fb5163e113d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fb5163e267f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fb5163e3ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fb5166d8b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2a0cafe (0x7fb50d08fafe in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2a0cc06 (0x7fb50d08fc06 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fb5171eac58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x35f0350 (0x7fb518a5d350 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x35f0ac9 (0x7fb518a5dac9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fb517224e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2ff562 (0x7fb52214b562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2ff956 (0x7fb52214b956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x55782e740c68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x55782e6fc499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x55782e6fc5fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x55782e6a84b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x55782e745098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x55782e6f2742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x55782e6aafaa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55782e746774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x55782e6f2742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x55782e6aafaa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xaa8dba (0x7fb5228f4dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fb5228f2ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fb5228f62d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fb5228f7b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fb519e9854c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fb5228f60c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4a249d3 (0x7fb519e919d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fb519e92568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fb519e8c867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x4a54552 (0x7fb519ec1552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fb50a1c690b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7fb539dd6bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7fb55a4316db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7fb55a15a61f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T02:58:47.0816717Z Traceback (most recent call last):
2022-11-23T02:58:47.0817125Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:58:47.0817355Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:58:47.0817810Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2022-11-23T02:58:47.0817972Z     return x.cpu() + y.cuda()
2022-11-23T02:58:47.0818256Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-11-23T02:58:47.0818561Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-11-23T02:58:47.0819163Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fb50a1d859b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:47.0819822Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fb50a1d3dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:47.0820414Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7fb5163e113d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0820914Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fb5163e267f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0821526Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fb5163e3ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0822086Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fb5166d8b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0822510Z frame #6: <unknown function> + 0x2a0cafe (0x7fb50d08fafe in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:47.0822934Z frame #7: <unknown function> + 0x2a0cc06 (0x7fb50d08fc06 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:47.0823523Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fb5171eac58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0823939Z frame #9: <unknown function> + 0x35f0350 (0x7fb518a5d350 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0824400Z frame #10: <unknown function> + 0x35f0ac9 (0x7fb518a5dac9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0824943Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fb517224e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0825355Z frame #12: <unknown function> + 0x2ff562 (0x7fb52214b562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0825782Z frame #13: <unknown function> + 0x2ff956 (0x7fb52214b956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0826016Z frame #14: <unknown function> + 0x1ddc68 (0x55782e740c68 in /opt/conda/bin/python)
2022-11-23T02:58:47.0826240Z frame #15: <unknown function> + 0x199499 (0x55782e6fc499 in /opt/conda/bin/python)
2022-11-23T02:58:47.0826463Z frame #16: <unknown function> + 0x1995fa (0x55782e6fc5fa in /opt/conda/bin/python)
2022-11-23T02:58:47.0826685Z frame #17: PyNumber_Add + 0x41 (0x55782e6a84b1 in /opt/conda/bin/python)
2022-11-23T02:58:47.0826927Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x55782e745098 in /opt/conda/bin/python)
2022-11-23T02:58:47.0827152Z frame #19: <unknown function> + 0x18f742 (0x55782e6f2742 in /opt/conda/bin/python)
2022-11-23T02:58:47.0827360Z frame #20: _PyObject_Call + 0x20a (0x55782e6aafaa in /opt/conda/bin/python)
2022-11-23T02:58:47.0827607Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55782e746774 in /opt/conda/bin/python)
2022-11-23T02:58:47.0827828Z frame #22: <unknown function> + 0x18f742 (0x55782e6f2742 in /opt/conda/bin/python)
2022-11-23T02:58:47.0828048Z frame #23: _PyObject_Call + 0x20a (0x55782e6aafaa in /opt/conda/bin/python)
2022-11-23T02:58:47.0828479Z frame #24: <unknown function> + 0xaa8dba (0x7fb5228f4dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0829046Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fb5228f2ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0829742Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fb5228f62d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0830526Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fb5228f7b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0831332Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fb519e9854c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0832156Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fb5228f60c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0832580Z frame #30: <unknown function> + 0x4a249d3 (0x7fb519e919d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0833330Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fb519e92568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0834047Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fb519e8c867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0834474Z frame #33: <unknown function> + 0x4a54552 (0x7fb519ec1552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0834917Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fb50a1c690b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:47.0835185Z frame #35: <unknown function> + 0xdbbf4 (0x7fb539dd6bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:58:47.0835546Z frame #36: <unknown function> + 0x76db (0x7fb55a4316db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:58:47.0835852Z frame #37: clone + 0x3f (0x7fb55a15a61f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:58:47.0835873Z 
2022-11-23T02:58:47.0835892Z 
2022-11-23T02:58:47.0836063Z On WorkerInfo(id=2, name=worker2):
2022-11-23T02:58:47.0848420Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f49a2be859b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f49a2be3dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7f49aedf113d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f49aedf267f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f49aedf3ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f49af0e8b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2a0cafe (0x7f49a5a9fafe in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2a0cc06 (0x7f49a5a9fc06 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f49afbfac58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x35f0350 (0x7f49b146d350 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x35f0ac9 (0x7f49b146dac9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f49afc34e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2ff562 (0x7f49bab5b562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2ff956 (0x7f49bab5b956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x557c33905c68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x557c338c1499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x557c338c15fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x557c3386d4b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x557c3390a098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x557c338b7742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x557c3386ffaa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x557c3390b774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x557c338b7742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x557c3386ffaa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xaa8dba (0x7f49bb304dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f49bb302ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f49bb3062d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f49bb307b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f49b28a854c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f49bb3060c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4a249d3 (0x7f49b28a19d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f49b28a2568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f49b289c867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x4a54552 (0x7f49b28d1552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f49a2bd690b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7f49d27e6bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7f49f2e416db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7f49f2b6a61f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T02:58:47.0848758Z Traceback (most recent call last):
2022-11-23T02:58:47.0849174Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:58:47.0849407Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:58:47.0849836Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2022-11-23T02:58:47.0850004Z     return x.cpu() + y.cuda()
2022-11-23T02:58:47.0850292Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-11-23T02:58:47.0850613Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-11-23T02:58:47.0851215Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f49a2be859b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:47.0851924Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f49a2be3dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:47.0852472Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7f49aedf113d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0852982Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f49aedf267f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0853598Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f49aedf3ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0854155Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f49af0e8b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0854577Z frame #6: <unknown function> + 0x2a0cafe (0x7f49a5a9fafe in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:47.0854981Z frame #7: <unknown function> + 0x2a0cc06 (0x7f49a5a9fc06 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:47.0855583Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f49afbfac58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0855994Z frame #9: <unknown function> + 0x35f0350 (0x7f49b146d350 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0856407Z frame #10: <unknown function> + 0x35f0ac9 (0x7f49b146dac9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0856944Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f49afc34e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0857369Z frame #12: <unknown function> + 0x2ff562 (0x7f49bab5b562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0857790Z frame #13: <unknown function> + 0x2ff956 (0x7f49bab5b956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0858074Z frame #14: <unknown function> + 0x1ddc68 (0x557c33905c68 in /opt/conda/bin/python)
2022-11-23T02:58:47.0858301Z frame #15: <unknown function> + 0x199499 (0x557c338c1499 in /opt/conda/bin/python)
2022-11-23T02:58:47.0858525Z frame #16: <unknown function> + 0x1995fa (0x557c338c15fa in /opt/conda/bin/python)
2022-11-23T02:58:47.0858731Z frame #17: PyNumber_Add + 0x41 (0x557c3386d4b1 in /opt/conda/bin/python)
2022-11-23T02:58:47.0858976Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x557c3390a098 in /opt/conda/bin/python)
2022-11-23T02:58:47.0859206Z frame #19: <unknown function> + 0x18f742 (0x557c338b7742 in /opt/conda/bin/python)
2022-11-23T02:58:47.0859431Z frame #20: _PyObject_Call + 0x20a (0x557c3386ffaa in /opt/conda/bin/python)
2022-11-23T02:58:47.0859673Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x557c3390b774 in /opt/conda/bin/python)
2022-11-23T02:58:47.0859897Z frame #22: <unknown function> + 0x18f742 (0x557c338b7742 in /opt/conda/bin/python)
2022-11-23T02:58:47.0860122Z frame #23: _PyObject_Call + 0x20a (0x557c3386ffaa in /opt/conda/bin/python)
2022-11-23T02:58:47.0860557Z frame #24: <unknown function> + 0xaa8dba (0x7f49bb304dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0861119Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f49bb302ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0861859Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f49bb3062d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0862573Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f49bb307b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0863382Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f49b28a854c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0864207Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f49bb3060c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0864629Z frame #30: <unknown function> + 0x4a249d3 (0x7f49b28a19d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0865337Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f49b28a2568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0866010Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f49b289c867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0866423Z frame #33: <unknown function> + 0x4a54552 (0x7f49b28d1552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0866884Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f49a2bd690b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:47.0867142Z frame #35: <unknown function> + 0xdbbf4 (0x7f49d27e6bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:58:47.0867500Z frame #36: <unknown function> + 0x76db (0x7f49f2e416db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:58:47.0867867Z frame #37: clone + 0x3f (0x7f49f2b6a61f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:58:47.0867888Z 
2022-11-23T02:58:47.0867906Z 
2022-11-23T02:58:47.0868080Z On WorkerInfo(id=3, name=worker3):
2022-11-23T02:58:47.0880371Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fe4a62e759b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fe4a62e2dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7fe4b24f013d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fe4b24f167f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fe4b24f2ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fe4b27e7b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2a0cafe (0x7fe4a919eafe in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2a0cc06 (0x7fe4a919ec06 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fe4b32f9c58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x35f0350 (0x7fe4b4b6c350 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x35f0ac9 (0x7fe4b4b6cac9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fe4b3333e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2ff562 (0x7fe4be25a562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2ff956 (0x7fe4be25a956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x5567190bac68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x556719076499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x5567190765fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x5567190224b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x5567190bf098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x55671906c742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x556719024faa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x5567190c0774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x55671906c742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x556719024faa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xaa8dba (0x7fe4bea03dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fe4bea01ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fe4bea052d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fe4bea06b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fe4b5fa754c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fe4bea050c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4a249d3 (0x7fe4b5fa09d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fe4b5fa1568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fe4b5f9b867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x4a54552 (0x7fe4b5fd0552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fe4a62d590b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7fe4d5ee5bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7fe4f65406db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7fe4f626961f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T02:58:47.0880668Z Traceback (most recent call last):
2022-11-23T02:58:47.0881075Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T02:58:47.0881306Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T02:58:47.0881751Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2022-11-23T02:58:47.0881912Z     return x.cpu() + y.cuda()
2022-11-23T02:58:47.0882200Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-11-23T02:58:47.0882527Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-11-23T02:58:47.0883127Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fe4a62e759b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:47.0883795Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fe4a62e2dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:47.0884331Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7fe4b24f013d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0884814Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fe4b24f167f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0885481Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fe4b24f2ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0886352Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fe4b27e7b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0886790Z frame #6: <unknown function> + 0x2a0cafe (0x7fe4a919eafe in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:47.0887212Z frame #7: <unknown function> + 0x2a0cc06 (0x7fe4a919ec06 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T02:58:47.0887808Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fe4b32f9c58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0888227Z frame #9: <unknown function> + 0x35f0350 (0x7fe4b4b6c350 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0888645Z frame #10: <unknown function> + 0x35f0ac9 (0x7fe4b4b6cac9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0889176Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fe4b3333e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0889688Z frame #12: <unknown function> + 0x2ff562 (0x7fe4be25a562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0890125Z frame #13: <unknown function> + 0x2ff956 (0x7fe4be25a956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0890343Z frame #14: <unknown function> + 0x1ddc68 (0x5567190bac68 in /opt/conda/bin/python)
2022-11-23T02:58:47.0890573Z frame #15: <unknown function> + 0x199499 (0x556719076499 in /opt/conda/bin/python)
2022-11-23T02:58:47.0890795Z frame #16: <unknown function> + 0x1995fa (0x5567190765fa in /opt/conda/bin/python)
2022-11-23T02:58:47.0891013Z frame #17: PyNumber_Add + 0x41 (0x5567190224b1 in /opt/conda/bin/python)
2022-11-23T02:58:47.0891255Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x5567190bf098 in /opt/conda/bin/python)
2022-11-23T02:58:47.0891479Z frame #19: <unknown function> + 0x18f742 (0x55671906c742 in /opt/conda/bin/python)
2022-11-23T02:58:47.0891700Z frame #20: _PyObject_Call + 0x20a (0x556719024faa in /opt/conda/bin/python)
2022-11-23T02:58:47.0891944Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x5567190c0774 in /opt/conda/bin/python)
2022-11-23T02:58:47.0892165Z frame #22: <unknown function> + 0x18f742 (0x55671906c742 in /opt/conda/bin/python)
2022-11-23T02:58:47.0892368Z frame #23: _PyObject_Call + 0x20a (0x556719024faa in /opt/conda/bin/python)
2022-11-23T02:58:47.0892806Z frame #24: <unknown function> + 0xaa8dba (0x7fe4bea03dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0893369Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fe4bea01ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0894069Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fe4bea052d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0894803Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fe4bea06b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0895613Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fe4b5fa754c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0896519Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fe4bea050c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T02:58:47.0896948Z frame #30: <unknown function> + 0x4a249d3 (0x7fe4b5fa09d3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0897655Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fe4b5fa1568 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0898329Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fe4b5f9b867 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0898751Z frame #33: <unknown function> + 0x4a54552 (0x7fe4b5fd0552 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T02:58:47.0899202Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fe4a62d590b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T02:58:47.0911971Z frame #35: <unknown function> + 0xdbbf4 (0x7fe4d5ee5bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T02:58:47.0912450Z frame #36: <unknown function> + 0x76db (0x7fe4f65406db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T02:58:47.0912779Z frame #37: clone + 0x3f (0x7fe4f626961f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T02:58:47.0912807Z 
2022-11-23T02:58:47.0912827Z 
2022-11-23T02:58:47.0912970Z ok (6.448s)
2022-11-23T02:58:47.0912989Z 
2022-11-23T02:58:47.0913308Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0913467Z Ran 1 test in 6.448s
2022-11-23T02:58:47.0913487Z 
2022-11-23T02:58:47.0913599Z OK
2022-11-23T02:58:47.0913639Z 
2022-11-23T02:58:47.0913791Z Generating XML reports...
2022-11-23T02:58:47.0914407Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025259.xml
2022-11-23T02:58:47.0914851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0915077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0915522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0915766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0916075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6wm08hph
2022-11-23T02:58:47.0916397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6wm08hph/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0916418Z 
2022-11-23T02:58:47.0916550Z Running tests...
2022-11-23T02:58:47.0916875Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0917298Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0917661Z   test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0917931Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104183
2022-11-23T02:58:47.0918204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104184
2022-11-23T02:58:47.0918541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104185
2022-11-23T02:58:47.0918797Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104186
2022-11-23T02:58:47.0919234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0919442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0919890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0920137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0920569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0920790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0921230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0921474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0921903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0922108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0922550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0922789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0923266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0923496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0923940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0924183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0924488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaec1e4gn
2022-11-23T02:58:47.0924807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaec1e4gn/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0925096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk12xupzi
2022-11-23T02:58:47.0925411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk12xupzi/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0926042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply0s_oo1
2022-11-23T02:58:47.0926373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply0s_oo1/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0926668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9re79r0
2022-11-23T02:58:47.0926981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9re79r0/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0927252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0927519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0927770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0928036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0928233Z fi_getinfo: -61
2022-11-23T02:58:47.0928412Z fi_getinfo: -61
2022-11-23T02:58:47.0928584Z fi_getinfo: -61
2022-11-23T02:58:47.0928759Z fi_getinfo: -61
2022-11-23T02:58:47.0928898Z ok (4.498s)
2022-11-23T02:58:47.0928919Z 
2022-11-23T02:58:47.0929224Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0929359Z Ran 1 test in 4.498s
2022-11-23T02:58:47.0929379Z 
2022-11-23T02:58:47.0929633Z OK
2022-11-23T02:58:47.0929654Z 
2022-11-23T02:58:47.0929822Z Generating XML reports...
2022-11-23T02:58:47.0930420Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025308.xml
2022-11-23T02:58:47.0930834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0931049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0931473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0931713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0931991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjg4cam7g
2022-11-23T02:58:47.0932302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjg4cam7g/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0932326Z 
2022-11-23T02:58:47.0932475Z Running tests...
2022-11-23T02:58:47.0932780Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0933224Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0933584Z   test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0933847Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104370
2022-11-23T02:58:47.0934175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104371
2022-11-23T02:58:47.0934442Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104372
2022-11-23T02:58:47.0934674Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104373
2022-11-23T02:58:47.0935097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0935316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0935726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0935939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0936358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0936586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0937009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0937222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0937628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0937850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0938264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0938490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0938899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0939106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0939521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0939748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0940028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4nnlpxfm
2022-11-23T02:58:47.0940321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0voji68z
2022-11-23T02:58:47.0940693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4nnlpxfm/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0940996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0voji68z/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0941290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwl1jr6gz
2022-11-23T02:58:47.0941596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwl1jr6gz/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0941888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpleaw3348
2022-11-23T02:58:47.0942196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpleaw3348/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0942446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0942716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0942984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0943247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0943437Z fi_getinfo: -61
2022-11-23T02:58:47.0943615Z fi_getinfo: -61
2022-11-23T02:58:47.0943793Z fi_getinfo: -61
2022-11-23T02:58:47.0943969Z fi_getinfo: -61
2022-11-23T02:58:47.0944092Z ok (4.528s)
2022-11-23T02:58:47.0944112Z 
2022-11-23T02:58:47.0944419Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0944570Z Ran 1 test in 4.528s
2022-11-23T02:58:47.0944589Z 
2022-11-23T02:58:47.0944766Z OK
2022-11-23T02:58:47.0944788Z 
2022-11-23T02:58:47.0944956Z Generating XML reports...
2022-11-23T02:58:47.0945551Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025316.xml
2022-11-23T02:58:47.0945961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0946180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0946584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0946816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0947106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dtcw8z6
2022-11-23T02:58:47.0947415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dtcw8z6/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0947435Z 
2022-11-23T02:58:47.0947586Z Running tests...
2022-11-23T02:58:47.0947889Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0948287Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0948659Z   test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0948920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104557
2022-11-23T02:58:47.0949159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104558
2022-11-23T02:58:47.0949413Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104559
2022-11-23T02:58:47.0949662Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104560
2022-11-23T02:58:47.0950082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0950301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0950722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0950955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0951423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0951618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0952038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0952268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0952678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0952892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0953303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0953531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0953941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0954151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0954544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0954770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0955066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl328tyy4
2022-11-23T02:58:47.0955420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl328tyy4/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0955724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedhwf21s
2022-11-23T02:58:47.0956035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedhwf21s/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0956332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuklm35vl
2022-11-23T02:58:47.0956637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuklm35vl/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0956929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58vpfraq
2022-11-23T02:58:47.0957213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58vpfraq/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0957481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0957752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0958017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0958279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0958470Z fi_getinfo: -61
2022-11-23T02:58:47.0958614Z ok (9.652s)
2022-11-23T02:58:47.0958634Z 
2022-11-23T02:58:47.0958937Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0959071Z Ran 1 test in 9.652s
2022-11-23T02:58:47.0959091Z 
2022-11-23T02:58:47.0959222Z OK
2022-11-23T02:58:47.0959242Z 
2022-11-23T02:58:47.0959402Z Generating XML reports...
2022-11-23T02:58:47.0959995Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025323.xml
2022-11-23T02:58:47.0960402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0960623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0961043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0961272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0961605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplajp50se
2022-11-23T02:58:47.0961914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplajp50se/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0961934Z 
2022-11-23T02:58:47.0962082Z Running tests...
2022-11-23T02:58:47.0962389Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0962787Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0963154Z   test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0963416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104772
2022-11-23T02:58:47.0963676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104773
2022-11-23T02:58:47.0963933Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104774
2022-11-23T02:58:47.0964172Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104775
2022-11-23T02:58:47.0964590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0964807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0965228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0965459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0966174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0966405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0966835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0967051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0967458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0967673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0968084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0968315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0968727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0968942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0969360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0969589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0969875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj64zqsqj
2022-11-23T02:58:47.0970186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj64zqsqj/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0970478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeq1zfxat
2022-11-23T02:58:47.0970786Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeq1zfxat/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0971076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3sbjmi3m
2022-11-23T02:58:47.0971384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3sbjmi3m/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0971674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcy8f3fye
2022-11-23T02:58:47.0971976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcy8f3fye/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0972320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0972568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0972836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0973101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0973288Z fi_getinfo: -61
2022-11-23T02:58:47.0973430Z ok (10.708s)
2022-11-23T02:58:47.0973450Z 
2022-11-23T02:58:47.0973756Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0973910Z Ran 1 test in 10.709s
2022-11-23T02:58:47.0973929Z 
2022-11-23T02:58:47.0974059Z OK
2022-11-23T02:58:47.0974078Z 
2022-11-23T02:58:47.0974224Z Generating XML reports...
2022-11-23T02:58:47.0974811Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025335.xml
2022-11-23T02:58:47.0975228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0975442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0975863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0976094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0976437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3oyzm9zh
2022-11-23T02:58:47.0976758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3oyzm9zh/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0976777Z 
2022-11-23T02:58:47.0976925Z Running tests...
2022-11-23T02:58:47.0977212Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0977613Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0977975Z   test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0978235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104988
2022-11-23T02:58:47.0978493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104989
2022-11-23T02:58:47.0978749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104990
2022-11-23T02:58:47.0979002Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104991
2022-11-23T02:58:47.0979416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0979614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0980037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0980273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0980676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0980893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0981303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0981534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0981937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0982132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0982541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0982825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0983235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0983446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0983852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0984078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0984380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq979tsd9
2022-11-23T02:58:47.0984689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq979tsd9/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0984964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmmp4v0r
2022-11-23T02:58:47.0985276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmmp4v0r/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0985565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2w8hujdt
2022-11-23T02:58:47.0985869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2w8hujdt/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0986159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6w16q57
2022-11-23T02:58:47.0986455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6w16q57/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0986768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.0987047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.0987315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.0987558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.0987756Z fi_getinfo: -61
2022-11-23T02:58:47.0987895Z ok (10.959s)
2022-11-23T02:58:47.0987914Z 
2022-11-23T02:58:47.0988217Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0988369Z Ran 1 test in 10.959s
2022-11-23T02:58:47.0988388Z 
2022-11-23T02:58:47.0988519Z OK
2022-11-23T02:58:47.0988538Z 
2022-11-23T02:58:47.0988703Z Generating XML reports...
2022-11-23T02:58:47.0989295Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025348.xml
2022-11-23T02:58:47.0989692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0989910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0990329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0990564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0990860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1e2b3hc
2022-11-23T02:58:47.0991168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1e2b3hc/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0991188Z 
2022-11-23T02:58:47.0991334Z Running tests...
2022-11-23T02:58:47.0991636Z ----------------------------------------------------------------------
2022-11-23T02:58:47.0992018Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.0992390Z   test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.0992651Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105204
2022-11-23T02:58:47.0992910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105205
2022-11-23T02:58:47.0993220Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105206
2022-11-23T02:58:47.0993471Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105207
2022-11-23T02:58:47.0993895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0994115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0994537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0994754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0995165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0995380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0995800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0996031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0996431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0996642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0997053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0997325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0997721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.0997935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.0998343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.0998573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.0998870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp211jhm4y
2022-11-23T02:58:47.0999182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp211jhm4y/_remote_module_non_scriptable.py
2022-11-23T02:58:47.0999475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8ajaay9
2022-11-23T02:58:47.0999781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8ajaay9/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1000056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpei_h8fvk
2022-11-23T02:58:47.1000362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpei_h8fvk/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1000654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv08gat4j
2022-11-23T02:58:47.1000964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv08gat4j/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1001232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1001498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1001764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1002027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1002198Z fi_getinfo: -61
2022-11-23T02:58:47.1002346Z ok (9.661s)
2022-11-23T02:58:47.1002366Z 
2022-11-23T02:58:47.1002672Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1002828Z Ran 1 test in 9.661s
2022-11-23T02:58:47.1002847Z 
2022-11-23T02:58:47.1002979Z OK
2022-11-23T02:58:47.1002998Z 
2022-11-23T02:58:47.1003166Z Generating XML reports...
2022-11-23T02:58:47.1003832Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025402.xml
2022-11-23T02:58:47.1004244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1004459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1004863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1005100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1005397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1rbtbkl
2022-11-23T02:58:47.1005926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1rbtbkl/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1005948Z 
2022-11-23T02:58:47.1006106Z Running tests...
2022-11-23T02:58:47.1006421Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1006822Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1007175Z   test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1007438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105419
2022-11-23T02:58:47.1007679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105420
2022-11-23T02:58:47.1008007Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105421
2022-11-23T02:58:47.1008269Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105422
2022-11-23T02:58:47.1008686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1008902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1009325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1009559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1009963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1010157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1010567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1010802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1011204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1011419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1011832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1012061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1012459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1012674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1013073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1013306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1013600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznbzxi7e
2022-11-23T02:58:47.1013909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznbzxi7e/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1014201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1d8sland
2022-11-23T02:58:47.1014579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1d8sland/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1014872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdpxu6n38
2022-11-23T02:58:47.1015179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdpxu6n38/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1015451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhcyak8l
2022-11-23T02:58:47.1015760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhcyak8l/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1016031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1016298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1016562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1016830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1017018Z fi_getinfo: -61
2022-11-23T02:58:47.1017197Z fi_getinfo: -61
2022-11-23T02:58:47.1017354Z fi_getinfo: -61
2022-11-23T02:58:47.1017530Z fi_getinfo: -61
2022-11-23T02:58:47.1017669Z ok (15.874s)
2022-11-23T02:58:47.1017689Z 
2022-11-23T02:58:47.1017990Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1018142Z Ran 1 test in 15.874s
2022-11-23T02:58:47.1018162Z 
2022-11-23T02:58:47.1018294Z OK
2022-11-23T02:58:47.1018313Z 
2022-11-23T02:58:47.1018523Z Generating XML reports...
2022-11-23T02:58:47.1019125Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025415.xml
2022-11-23T02:58:47.1019517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1019738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1020159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1020391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1020685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2nojrmn
2022-11-23T02:58:47.1020991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2nojrmn/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1021012Z 
2022-11-23T02:58:47.1021161Z Running tests...
2022-11-23T02:58:47.1021465Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1021847Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1022203Z   test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1022467Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105766
2022-11-23T02:58:47.1022722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105767
2022-11-23T02:58:47.1022977Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105768
2022-11-23T02:58:47.1023229Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105769
2022-11-23T02:58:47.1023646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1023867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1024290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1024504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1024907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1025172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1025583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1025813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1026210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1026425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1026835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1027063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1027455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1027674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1028086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1028314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1028608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9efqlv90
2022-11-23T02:58:47.1028919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9efqlv90/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1029257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi23ij8lw
2022-11-23T02:58:47.1029568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi23ij8lw/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1029845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzf3bkbow
2022-11-23T02:58:47.1030157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzf3bkbow/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1030448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpua7en00r
2022-11-23T02:58:47.1030754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpua7en00r/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1031022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1031288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1031560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1031826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1032018Z fi_getinfo: -61
2022-11-23T02:58:47.1032179Z fi_getinfo: -61
2022-11-23T02:58:47.1032354Z fi_getinfo: -61
2022-11-23T02:58:47.1032534Z fi_getinfo: -61
2022-11-23T02:58:47.1032672Z ok (17.417s)
2022-11-23T02:58:47.1032692Z 
2022-11-23T02:58:47.1032996Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1033188Z Ran 1 test in 17.417s
2022-11-23T02:58:47.1033208Z 
2022-11-23T02:58:47.1033341Z OK
2022-11-23T02:58:47.1033360Z 
2022-11-23T02:58:47.1033505Z Generating XML reports...
2022-11-23T02:58:47.1034097Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025433.xml
2022-11-23T02:58:47.1034515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1034733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1035154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1035386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1035734Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8znu23r
2022-11-23T02:58:47.1036039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8znu23r/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1036058Z 
2022-11-23T02:58:47.1036211Z Running tests...
2022-11-23T02:58:47.1036503Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1036904Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1037259Z   test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1038074Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81962 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.617s)
2022-11-23T02:58:47.1038097Z 
2022-11-23T02:58:47.1038398Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1038550Z Ran 1 test in 1.617s
2022-11-23T02:58:47.1038570Z 
2022-11-23T02:58:47.1038720Z OK (skipped=1)
2022-11-23T02:58:47.1038739Z 
2022-11-23T02:58:47.1038904Z Generating XML reports...
2022-11-23T02:58:47.1039493Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025453.xml
2022-11-23T02:58:47.1039935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1040157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1040579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1040814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1041109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ycvfcjx
2022-11-23T02:58:47.1041417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ycvfcjx/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1041437Z 
2022-11-23T02:58:47.1041583Z Running tests...
2022-11-23T02:58:47.1041886Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1042282Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1042620Z   test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1042880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106153
2022-11-23T02:58:47.1043139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106154
2022-11-23T02:58:47.1043401Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106155
2022-11-23T02:58:47.1043649Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106156
2022-11-23T02:58:47.1044069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1044283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1044707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1044940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1045334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1045553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1046190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1046505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1046912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1047125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1047534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1047761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1048156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1048372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1048779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1049012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1049309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcdhmizvm
2022-11-23T02:58:47.1049619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcdhmizvm/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1049914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmnoyuiy
2022-11-23T02:58:47.1050224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmnoyuiy/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1050575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3hjn405d
2022-11-23T02:58:47.1050871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3hjn405d/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1051162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjbpc3yth
2022-11-23T02:58:47.1051467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjbpc3yth/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1051741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1052007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1052272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1052533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1052723Z fi_getinfo: -61
2022-11-23T02:58:47.1052883Z fi_getinfo: -61
2022-11-23T02:58:47.1053062Z fi_getinfo: -61
2022-11-23T02:58:47.1053237Z fi_getinfo: -61
2022-11-23T02:58:47.1053378Z ok (17.330s)
2022-11-23T02:58:47.1053397Z 
2022-11-23T02:58:47.1053703Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1053859Z Ran 1 test in 17.330s
2022-11-23T02:58:47.1053878Z 
2022-11-23T02:58:47.1054008Z OK
2022-11-23T02:58:47.1054030Z 
2022-11-23T02:58:47.1054175Z Generating XML reports...
2022-11-23T02:58:47.1054765Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025457.xml
2022-11-23T02:58:47.1055177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1055397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1055818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1056053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1056346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1t19yu4e
2022-11-23T02:58:47.1056651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1t19yu4e/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1056767Z 
2022-11-23T02:58:47.1056920Z Running tests...
2022-11-23T02:58:47.1057206Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1057609Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1057958Z   test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1058216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106506
2022-11-23T02:58:47.1058475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106507
2022-11-23T02:58:47.1058734Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106508
2022-11-23T02:58:47.1058986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106509
2022-11-23T02:58:47.1059403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1059626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1060026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1060260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1060661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1060871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1061324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1061563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1061965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1062182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1062569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1062786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1063202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1063432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1063844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1064071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1064365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvpubu4j
2022-11-23T02:58:47.1064676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvpubu4j/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1064974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvy61wdtp
2022-11-23T02:58:47.1065264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvy61wdtp/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1065556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg5rdprfe
2022-11-23T02:58:47.1065863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg5rdprfe/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1066150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcso_jck
2022-11-23T02:58:47.1066455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcso_jck/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1066721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1066987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1067308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1067557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1067752Z fi_getinfo: -61
2022-11-23T02:58:47.1067932Z fi_getinfo: -61
2022-11-23T02:58:47.1068112Z fi_getinfo: -61
2022-11-23T02:58:47.1068289Z fi_getinfo: -61
2022-11-23T02:58:47.1068429Z ok (15.963s)
2022-11-23T02:58:47.1068449Z 
2022-11-23T02:58:47.1068753Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1068888Z Ran 1 test in 15.963s
2022-11-23T02:58:47.1068925Z 
2022-11-23T02:58:47.1069042Z OK
2022-11-23T02:58:47.1069062Z 
2022-11-23T02:58:47.1069225Z Generating XML reports...
2022-11-23T02:58:47.1069818Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025518.xml
2022-11-23T02:58:47.1070229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1070450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1070873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1071103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1071396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcr_gzeol
2022-11-23T02:58:47.1071688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcr_gzeol/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1071784Z 
2022-11-23T02:58:47.1071920Z Running tests...
2022-11-23T02:58:47.1072233Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1072632Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1072992Z   test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1073249Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106853
2022-11-23T02:58:47.1073507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106854
2022-11-23T02:58:47.1073762Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106855
2022-11-23T02:58:47.1074014Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106856
2022-11-23T02:58:47.1074416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1074634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1075052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1075285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1075694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1075906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1076319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1076549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1076931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1077149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1077563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1077787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1078265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1078478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1078890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1079114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1079411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnn027m2b
2022-11-23T02:58:47.1079705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnn027m2b/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1080005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpovysbghh
2022-11-23T02:58:47.1080312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpovysbghh/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1080601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4swo4dkb
2022-11-23T02:58:47.1080909Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4swo4dkb/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1081195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19wvc3jw
2022-11-23T02:58:47.1081498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19wvc3jw/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1081764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1082063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1082337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1082601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1082790Z fi_getinfo: -61
2022-11-23T02:58:47.1082969Z fi_getinfo: -61
2022-11-23T02:58:47.1083152Z fi_getinfo: -61
2022-11-23T02:58:47.1083327Z fi_getinfo: -61
2022-11-23T02:58:47.1083449Z ok (15.143s)
2022-11-23T02:58:47.1083487Z 
2022-11-23T02:58:47.1083775Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1083927Z Ran 1 test in 15.143s
2022-11-23T02:58:47.1083946Z 
2022-11-23T02:58:47.1084078Z OK
2022-11-23T02:58:47.1084098Z 
2022-11-23T02:58:47.1084260Z Generating XML reports...
2022-11-23T02:58:47.1084849Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025536.xml
2022-11-23T02:58:47.1085271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1085488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1086126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1086351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1086651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidxryw_4
2022-11-23T02:58:47.1086967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidxryw_4/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1086987Z 
2022-11-23T02:58:47.1087135Z Running tests...
2022-11-23T02:58:47.1087441Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1087839Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1088199Z   test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1088460Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107199
2022-11-23T02:58:47.1088721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107200
2022-11-23T02:58:47.1089043Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107201
2022-11-23T02:58:47.1089295Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107202
2022-11-23T02:58:47.1089717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1089934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1090345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1090564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1090982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1091210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1091612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1091848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1092256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1092469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1092878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1093108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1093572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1093795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1094208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1094420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1094717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3u0z6e_
2022-11-23T02:58:47.1095025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3u0z6e_/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1095320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7brtlpqe
2022-11-23T02:58:47.1095628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7brtlpqe/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1095922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplk5y3jml
2022-11-23T02:58:47.1096226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplk5y3jml/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1096517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2e_75o9m
2022-11-23T02:58:47.1096804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2e_75o9m/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1097074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1097341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1097609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1097873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1098061Z fi_getinfo: -61
2022-11-23T02:58:47.1098241Z fi_getinfo: -61
2022-11-23T02:58:47.1098418Z fi_getinfo: -61
2022-11-23T02:58:47.1098579Z fi_getinfo: -61
2022-11-23T02:58:47.1098717Z ok (15.566s)
2022-11-23T02:58:47.1098736Z 
2022-11-23T02:58:47.1099040Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1099237Z Ran 1 test in 15.566s
2022-11-23T02:58:47.1099306Z 
2022-11-23T02:58:47.1099443Z OK
2022-11-23T02:58:47.1099463Z 
2022-11-23T02:58:47.1099633Z Generating XML reports...
2022-11-23T02:58:47.1100231Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025554.xml
2022-11-23T02:58:47.1100647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1100846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1101270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1101502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1101797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxy0_de4
2022-11-23T02:58:47.1102102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxy0_de4/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1102124Z 
2022-11-23T02:58:47.1102273Z Running tests...
2022-11-23T02:58:47.1102578Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1102974Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1103330Z   test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1103574Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107548
2022-11-23T02:58:47.1103881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107549
2022-11-23T02:58:47.1104144Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107550
2022-11-23T02:58:47.1104396Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107551
2022-11-23T02:58:47.1104812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1105034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1105454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1105686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1106075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1106294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1106710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1106939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1107342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1107556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1107965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1108197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1108596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1108792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1109214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1109443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1109742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphuzjkxlj
2022-11-23T02:58:47.1110055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphuzjkxlj/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1110407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiw6khn1c
2022-11-23T02:58:47.1110717Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiw6khn1c/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1111009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhk7cg8x
2022-11-23T02:58:47.1111296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhk7cg8x/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1111593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfh576ys2
2022-11-23T02:58:47.1111895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfh576ys2/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1112161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1112426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1112695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1112959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1113148Z fi_getinfo: -61
2022-11-23T02:58:47.1113309Z fi_getinfo: -61
2022-11-23T02:58:47.1113490Z fi_getinfo: -61
2022-11-23T02:58:47.1113663Z fi_getinfo: -61
2022-11-23T02:58:47.1113801Z ok (15.763s)
2022-11-23T02:58:47.1113821Z 
2022-11-23T02:58:47.1114125Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1114321Z Ran 1 test in 15.763s
2022-11-23T02:58:47.1114343Z 
2022-11-23T02:58:47.1114477Z OK
2022-11-23T02:58:47.1114497Z 
2022-11-23T02:58:47.1114667Z Generating XML reports...
2022-11-23T02:58:47.1115240Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025612.xml
2022-11-23T02:58:47.1115660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1115878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1116298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1116528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1116822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsonr4ays
2022-11-23T02:58:47.1117136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsonr4ays/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1117155Z 
2022-11-23T02:58:47.1117304Z Running tests...
2022-11-23T02:58:47.1117607Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1117987Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1118352Z   test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1118611Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107897
2022-11-23T02:58:47.1118866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107898
2022-11-23T02:58:47.1119121Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107899
2022-11-23T02:58:47.1119375Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107900
2022-11-23T02:58:47.1119794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1120011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1120413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1120701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1121113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1121329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1121742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1121971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1122373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1122585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1122999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1123208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1123621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1123829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1124236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1124461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1124760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1ylc9_h
2022-11-23T02:58:47.1125117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1ylc9_h/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1125418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cum_43e
2022-11-23T02:58:47.1126003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cum_43e/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1126318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt656joeb
2022-11-23T02:58:47.1126627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt656joeb/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1126922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpciekjnjl
2022-11-23T02:58:47.1127226Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpciekjnjl/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1127493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1127767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1128035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1128298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1128473Z fi_getinfo: -61
2022-11-23T02:58:47.1128662Z fi_getinfo: -61
2022-11-23T02:58:47.1128839Z fi_getinfo: -61
2022-11-23T02:58:47.1129014Z fi_getinfo: -61
2022-11-23T02:58:47.1129154Z ok (15.522s)
2022-11-23T02:58:47.1129174Z 
2022-11-23T02:58:47.1129477Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1129630Z Ran 1 test in 15.523s
2022-11-23T02:58:47.1129649Z 
2022-11-23T02:58:47.1129761Z OK
2022-11-23T02:58:47.1129782Z 
2022-11-23T02:58:47.1129946Z Generating XML reports...
2022-11-23T02:58:47.1130541Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025631.xml
2022-11-23T02:58:47.1130956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1131173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1131593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1131909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1132205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1nxqfd87
2022-11-23T02:58:47.1132514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1nxqfd87/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1132534Z 
2022-11-23T02:58:47.1132665Z Running tests...
2022-11-23T02:58:47.1132973Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1133413Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1133771Z   test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1134032Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108243
2022-11-23T02:58:47.1134290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108244
2022-11-23T02:58:47.1134550Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108245
2022-11-23T02:58:47.1134802Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108246
2022-11-23T02:58:47.1135199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1135416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1135894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1136135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1136546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1136758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1137173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1137388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1137808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1138020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1138433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1138664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1139069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1139282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1139692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1139924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1140221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadquwgsk
2022-11-23T02:58:47.1140514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadquwgsk/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1140810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfii1kqe
2022-11-23T02:58:47.1141121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfii1kqe/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1141412Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxclewzpi
2022-11-23T02:58:47.1141718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxclewzpi/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1142007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal53n5w8
2022-11-23T02:58:47.1142367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal53n5w8/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1142635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1142901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1143149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1143413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1143610Z fi_getinfo: -61
2022-11-23T02:58:47.1143789Z fi_getinfo: -61
2022-11-23T02:58:47.1143964Z fi_getinfo: -61
2022-11-23T02:58:47.1144140Z fi_getinfo: -61
2022-11-23T02:58:47.1144281Z ok (15.950s)
2022-11-23T02:58:47.1144301Z 
2022-11-23T02:58:47.1144586Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1144746Z Ran 1 test in 15.950s
2022-11-23T02:58:47.1144764Z 
2022-11-23T02:58:47.1144897Z OK
2022-11-23T02:58:47.1144916Z 
2022-11-23T02:58:47.1145083Z Generating XML reports...
2022-11-23T02:58:47.1145674Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025649.xml
2022-11-23T02:58:47.1146088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1146304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1146772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1147011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1147289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7tp8a8v
2022-11-23T02:58:47.1147598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7tp8a8v/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1147622Z 
2022-11-23T02:58:47.1147770Z Running tests...
2022-11-23T02:58:47.1148077Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1148474Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1148829Z   test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1149089Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108590
2022-11-23T02:58:47.1149351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108591
2022-11-23T02:58:47.1149591Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108592
2022-11-23T02:58:47.1149841Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108593
2022-11-23T02:58:47.1150262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1150476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1150893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1151124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1151528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1151743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1152157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1152370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1152776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1153058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1153481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1153708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1154111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1154324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1154741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1154948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1155248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8dq8j58
2022-11-23T02:58:47.1155562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8dq8j58/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1155855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvvoff4k
2022-11-23T02:58:47.1156163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvvoff4k/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1156452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpio_u8boi
2022-11-23T02:58:47.1156752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpio_u8boi/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1157088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lixoy6q
2022-11-23T02:58:47.1157398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lixoy6q/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1157648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1157918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1158184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1158449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1158636Z fi_getinfo: -61
2022-11-23T02:58:47.1158810Z fi_getinfo: -61
2022-11-23T02:58:47.1158985Z fi_getinfo: -61
2022-11-23T02:58:47.1159141Z fi_getinfo: -61
2022-11-23T02:58:47.1159282Z ok (17.462s)
2022-11-23T02:58:47.1159301Z 
2022-11-23T02:58:47.1159607Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1159762Z Ran 1 test in 17.462s
2022-11-23T02:58:47.1159781Z 
2022-11-23T02:58:47.1159911Z OK
2022-11-23T02:58:47.1159930Z 
2022-11-23T02:58:47.1160093Z Generating XML reports...
2022-11-23T02:58:47.1160683Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025708.xml
2022-11-23T02:58:47.1161100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1161316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1161716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1161945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1162241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaa800mup
2022-11-23T02:58:47.1162553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaa800mup/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1162573Z 
2022-11-23T02:58:47.1162721Z Running tests...
2022-11-23T02:58:47.1163019Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1163416Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1163829Z   test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1164092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108943
2022-11-23T02:58:47.1164331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108944
2022-11-23T02:58:47.1164586Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108945
2022-11-23T02:58:47.1164837Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108946
2022-11-23T02:58:47.1165257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1165470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1166115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1166358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1166772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1166966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1167374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1167587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1168070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1168310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1168729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1168962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1169367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1169581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1169973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1170204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1170505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgyroo745
2022-11-23T02:58:47.1170815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgyroo745/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1171109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegu2yr3l
2022-11-23T02:58:47.1171415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegu2yr3l/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1171710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjuiwfjuo
2022-11-23T02:58:47.1172017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjuiwfjuo/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1172289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporx6kd03
2022-11-23T02:58:47.1172589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporx6kd03/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1172858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1173125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1173390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1173653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1173909Z fi_getinfo: -61
2022-11-23T02:58:47.1174086Z fi_getinfo: -61
2022-11-23T02:58:47.1174243Z fi_getinfo: -61
2022-11-23T02:58:47.1174420Z fi_getinfo: -61
2022-11-23T02:58:47.1174561Z ok (16.210s)
2022-11-23T02:58:47.1174581Z 
2022-11-23T02:58:47.1174884Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1175036Z Ran 1 test in 16.210s
2022-11-23T02:58:47.1175055Z 
2022-11-23T02:58:47.1175182Z OK
2022-11-23T02:58:47.1175202Z 
2022-11-23T02:58:47.1175365Z Generating XML reports...
2022-11-23T02:58:47.1175955Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025728.xml
2022-11-23T02:58:47.1176349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1176564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1176987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1177216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1177510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgmi7jn7
2022-11-23T02:58:47.1177818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgmi7jn7/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1177838Z 
2022-11-23T02:58:47.1177984Z Running tests...
2022-11-23T02:58:47.1178283Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1178709Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1179070Z   test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1179330Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109290
2022-11-23T02:58:47.1179590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109291
2022-11-23T02:58:47.1179846Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109292
2022-11-23T02:58:47.1180098Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109293
2022-11-23T02:58:47.1180513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1180728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1181155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1181368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1181770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1181981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1182398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1182626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1183024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1183235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1183648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1183875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1184267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1184481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1184949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1185173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1185470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1yt_p3q
2022-11-23T02:58:47.1185780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1yt_p3q/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1186078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0dhlnimt
2022-11-23T02:58:47.1186388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0dhlnimt/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1186661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsv4zyduj
2022-11-23T02:58:47.1186968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsv4zyduj/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1187260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_eo1oxgn
2022-11-23T02:58:47.1187563Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_eo1oxgn/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1187830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1188092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1188356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1188667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1188863Z fi_getinfo: -61
2022-11-23T02:58:47.1189025Z fi_getinfo: -61
2022-11-23T02:58:47.1189200Z fi_getinfo: -61
2022-11-23T02:58:47.1189377Z fi_getinfo: -61
2022-11-23T02:58:47.1189515Z ok (17.066s)
2022-11-23T02:58:47.1189535Z 
2022-11-23T02:58:47.1189837Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1189996Z Ran 1 test in 17.066s
2022-11-23T02:58:47.1190014Z 
2022-11-23T02:58:47.1190144Z OK
2022-11-23T02:58:47.1190167Z 
2022-11-23T02:58:47.1190312Z Generating XML reports...
2022-11-23T02:58:47.1190903Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025747.xml
2022-11-23T02:58:47.1191312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1191528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1191951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1192182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1192476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg1lw5zyd
2022-11-23T02:58:47.1192785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg1lw5zyd/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1192805Z 
2022-11-23T02:58:47.1192953Z Running tests...
2022-11-23T02:58:47.1193240Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1193638Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1193998Z   test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1194264Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109643
2022-11-23T02:58:47.1194522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109644
2022-11-23T02:58:47.1194777Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109645
2022-11-23T02:58:47.1195027Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109646
2022-11-23T02:58:47.1195499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1195695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1196117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1196351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1196759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1196976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1197389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1197618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1198022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1198236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1198627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1198854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1199252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1199509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1199929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1200153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1200447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29z5xyx6
2022-11-23T02:58:47.1200758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29z5xyx6/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1201034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8dh3q443
2022-11-23T02:58:47.1201340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8dh3q443/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1201632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoy56aia0
2022-11-23T02:58:47.1201941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoy56aia0/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1202234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp16gri2qo
2022-11-23T02:58:47.1202535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp16gri2qo/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1202801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1203073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1203337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1203585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1203775Z fi_getinfo: -61
2022-11-23T02:58:47.1203955Z fi_getinfo: -61
2022-11-23T02:58:47.1204129Z fi_getinfo: -61
2022-11-23T02:58:47.1204304Z fi_getinfo: -61
2022-11-23T02:58:47.1204448Z ok (6.533s)
2022-11-23T02:58:47.1204467Z 
2022-11-23T02:58:47.1204774Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1204909Z Ran 1 test in 6.533s
2022-11-23T02:58:47.1204928Z 
2022-11-23T02:58:47.1205062Z OK
2022-11-23T02:58:47.1205082Z 
2022-11-23T02:58:47.1205248Z Generating XML reports...
2022-11-23T02:58:47.1206056Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025806.xml
2022-11-23T02:58:47.1206561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1206778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1207202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1207433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1207732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ji2thdt
2022-11-23T02:58:47.1208024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ji2thdt/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1208044Z 
2022-11-23T02:58:47.1208191Z Running tests...
2022-11-23T02:58:47.1208494Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1208895Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1209243Z   test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1209503Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109994
2022-11-23T02:58:47.1209762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109995
2022-11-23T02:58:47.1210017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109996
2022-11-23T02:58:47.1210313Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109997
2022-11-23T02:58:47.1210741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1210956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1211378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1211613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1212017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1212229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1212641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1212871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1213254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1213466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1213869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1214086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1214504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1214733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1215151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1215380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1215664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe32jzw9v
2022-11-23T02:58:47.1215974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe32jzw9v/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1216266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbk1kdkj1
2022-11-23T02:58:47.1216629Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbk1kdkj1/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1216918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf97d9oj
2022-11-23T02:58:47.1217221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf97d9oj/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1217512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0hgb2zs
2022-11-23T02:58:47.1217815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0hgb2zs/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1218085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1218338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1218602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1218865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1219060Z fi_getinfo: -61
2022-11-23T02:58:47.1219238Z fi_getinfo: -61
2022-11-23T02:58:47.1219417Z fi_getinfo: -61
2022-11-23T02:58:47.1219592Z fi_getinfo: -61
2022-11-23T02:58:47.1219715Z ok (8.550s)
2022-11-23T02:58:47.1219734Z 
2022-11-23T02:58:47.1220038Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1220187Z Ran 1 test in 8.550s
2022-11-23T02:58:47.1220207Z 
2022-11-23T02:58:47.1220342Z OK
2022-11-23T02:58:47.1220362Z 
2022-11-23T02:58:47.1220522Z Generating XML reports...
2022-11-23T02:58:47.1221192Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025816.xml
2022-11-23T02:58:47.1221614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1221829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1222257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1222468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1222763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpez75e40i
2022-11-23T02:58:47.1223071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpez75e40i/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1223091Z 
2022-11-23T02:58:47.1223239Z Running tests...
2022-11-23T02:58:47.1223547Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1223949Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1224313Z   test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1224573Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110641
2022-11-23T02:58:47.1224816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110642
2022-11-23T02:58:47.1225074Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110643
2022-11-23T02:58:47.1225323Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110644
2022-11-23T02:58:47.1225739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1225955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1226375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1226607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1227013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1227292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1227694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1227924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1228321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1228533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1228942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1229170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1229576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1229792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1230182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1230410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1230706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjek5ord2
2022-11-23T02:58:47.1231014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjek5ord2/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1231308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk39xhgaz
2022-11-23T02:58:47.1231661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk39xhgaz/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1231960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3576wot
2022-11-23T02:58:47.1232260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3576wot/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1232557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxfo74rd
2022-11-23T02:58:47.1232843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxfo74rd/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1233153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1233423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1233686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1233955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1234144Z skip: Need at least 4 CUDA devices (4.448s)
2022-11-23T02:58:47.1234164Z 
2022-11-23T02:58:47.1234476Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1234632Z Ran 1 test in 4.448s
2022-11-23T02:58:47.1234654Z 
2022-11-23T02:58:47.1234784Z OK (skipped=1)
2022-11-23T02:58:47.1234821Z 
2022-11-23T02:58:47.1234966Z Generating XML reports...
2022-11-23T02:58:47.1235575Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123025827.xml
2022-11-23T02:58:47.1235981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1236196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1236618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1236847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1237140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyv_oivzl
2022-11-23T02:58:47.1237451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyv_oivzl/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1237518Z 
2022-11-23T02:58:47.1237653Z Running tests...
2022-11-23T02:58:47.1237958Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1238357Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1238721Z   test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1238980Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110812
2022-11-23T02:58:47.1239243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110813
2022-11-23T02:58:47.1239500Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110814
2022-11-23T02:58:47.1239753Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110815
2022-11-23T02:58:47.1240169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1240369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1240790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1241019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1241421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1241632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1242099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1242337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1242738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1242935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1243347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1243575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1243980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1244193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1244603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1244830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1245128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp901z8apo
2022-11-23T02:58:47.1245441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp901z8apo/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1245939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphi0f0nse
2022-11-23T02:58:47.1246260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphi0f0nse/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1246554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnmwuetuz
2022-11-23T02:58:47.1246866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnmwuetuz/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1247160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpia7_l5zh
2022-11-23T02:58:47.1247461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpia7_l5zh/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1247729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1247996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1248333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1248598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1248788Z skip: Need at least 4 CUDA devices (4.443s)
2022-11-23T02:58:47.1248807Z 
2022-11-23T02:58:47.1249120Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1249274Z Ran 1 test in 4.443s
2022-11-23T02:58:47.1249293Z 
2022-11-23T02:58:47.1249440Z OK (skipped=1)
2022-11-23T02:58:47.1249460Z 
2022-11-23T02:58:47.1249626Z Generating XML reports...
2022-11-23T02:58:47.1250236Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123025834.xml
2022-11-23T02:58:47.1250647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1250847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1251268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1251499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1251795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmjdp17d
2022-11-23T02:58:47.1252104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmjdp17d/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1252123Z 
2022-11-23T02:58:47.1252331Z Running tests...
2022-11-23T02:58:47.1252640Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1253040Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T02:58:47.1253407Z   test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:58:47.1253653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110983
2022-11-23T02:58:47.1253911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110984
2022-11-23T02:58:47.1254167Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 110985
2022-11-23T02:58:47.1254415Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 110986
2022-11-23T02:58:47.1254829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1255048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1255456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1255669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1256076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1256311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1256728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1256955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1257356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1257576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1257994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1258224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1258633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:58:47.1258883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:58:47.1259301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:58:47.1259530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:58:47.1259826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2t_qm91t
2022-11-23T02:58:47.1260134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2t_qm91t/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1260434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprp7r6n30
2022-11-23T02:58:47.1260742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprp7r6n30/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1261032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ydbgsst
2022-11-23T02:58:47.1261321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ydbgsst/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1261614Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprl2cep3n
2022-11-23T02:58:47.1261919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprl2cep3n/_remote_module_non_scriptable.py
2022-11-23T02:58:47.1262187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:58:47.1262452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:58:47.1262763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:58:47.1263038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:58:47.1263227Z skip: Need at least 4 CUDA devices (4.455s)
2022-11-23T02:58:47.1263247Z 
2022-11-23T02:58:47.1263557Z ----------------------------------------------------------------------
2022-11-23T02:58:47.1263696Z Ran 1 test in 4.456s
2022-11-23T02:58:47.1263715Z 
2022-11-23T02:58:47.1263862Z OK (skipped=1)
2022-11-23T02:58:47.1263882Z 
2022-11-23T02:58:47.1264047Z Generating XML reports...
2022-11-23T02:58:47.1264659Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123025841.xml
2022-11-23T02:58:47.1264679Z 
2022-11-23T02:58:47.1265249Z ##[endgroup]
2022-11-23T02:58:47.1265808Z FINISHED PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_lf2dvlz7)
2022-11-23T02:58:47.1265829Z 
2022-11-23T02:58:47.2422156Z 
2022-11-23T02:58:47.2422492Z real	99m47.864s
2022-11-23T02:58:47.2423529Z user	172m45.273s
2022-11-23T02:58:47.2423935Z sys	90m29.987s
2022-11-23T02:58:47.2424241Z + assert_git_not_dirty
2022-11-23T02:58:47.2424852Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *rocm* ]]
2022-11-23T02:58:47.2425161Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *xla* ]]
2022-11-23T02:58:47.2426130Z ++ git status --porcelain
2022-11-23T02:58:48.8943927Z + git_status=
2022-11-23T02:58:48.8944849Z + [[ -n '' ]]
2022-11-23T02:58:48.8945571Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *cuda* ]]
2022-11-23T02:58:48.8945935Z + [[ 1 == 1 ]]
2022-11-23T02:58:48.8946326Z + echo 'Testing distributed C++ tests'
2022-11-23T02:58:48.8946697Z Testing distributed C++ tests
2022-11-23T02:58:48.8948132Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T02:58:48.8959848Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T02:58:48.8970525Z + TEST_REPORTS_DIR=test/test-reports/cpp-distributed/test_distributed
2022-11-23T02:58:48.8971501Z + mkdir -p test/test-reports/cpp-distributed/test_distributed
2022-11-23T02:58:48.8985381Z + /opt/conda/lib/python3.10/site-packages/torch/bin/FileStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml
2022-11-23T02:58:49.2259328Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-11-23T02:58:49.2260642Z [==========] Running 4 tests from 1 test suite.
2022-11-23T02:58:49.2261152Z [----------] Global test environment set-up.
2022-11-23T02:58:49.2261635Z [----------] 4 tests from FileStoreTest
2022-11-23T02:58:49.2262080Z [ RUN      ] FileStoreTest.testGetAndSet
2022-11-23T02:58:49.2265549Z [       OK ] FileStoreTest.testGetAndSet (0 ms)
2022-11-23T02:58:49.2266563Z [ RUN      ] FileStoreTest.testGetAndSetWithPrefix
2022-11-23T02:58:49.2271003Z [       OK ] FileStoreTest.testGetAndSetWithPrefix (0 ms)
2022-11-23T02:58:49.2272024Z [ RUN      ] FileStoreTest.testStressStore
2022-11-23T02:58:49.2470327Z [       OK ] FileStoreTest.testStressStore (19 ms)
2022-11-23T02:58:49.2471375Z [ RUN      ] FileStoreTest.testStressStoreWithPrefix
2022-11-23T02:58:49.2670808Z [       OK ] FileStoreTest.testStressStoreWithPrefix (20 ms)
2022-11-23T02:58:49.2672006Z [----------] 4 tests from FileStoreTest (41 ms total)
2022-11-23T02:58:49.2672516Z 
2022-11-23T02:58:49.2672893Z [----------] Global test environment tear-down
2022-11-23T02:58:49.2673400Z [==========] 4 tests from 1 test suite ran. (41 ms total)
2022-11-23T02:58:49.2673810Z [  PASSED  ] 4 tests.
2022-11-23T02:58:49.3327588Z + /opt/conda/lib/python3.10/site-packages/torch/bin/HashStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml
2022-11-23T02:58:49.6627425Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-11-23T02:58:49.6628707Z [==========] Running 4 tests from 1 test suite.
2022-11-23T02:58:49.6629187Z [----------] Global test environment set-up.
2022-11-23T02:58:49.6629679Z [----------] 4 tests from HashStoreTest
2022-11-23T02:58:49.6630144Z [ RUN      ] HashStoreTest.testGetAndSet
2022-11-23T02:58:49.7630691Z [       OK ] HashStoreTest.testGetAndSet (100 ms)
2022-11-23T02:58:49.7631735Z [ RUN      ] HashStoreTest.testGetAndSetWithPrefix
2022-11-23T02:58:49.8633051Z [       OK ] HashStoreTest.testGetAndSetWithPrefix (100 ms)
2022-11-23T02:58:49.8634044Z [ RUN      ] HashStoreTest.testStressStore
2022-11-23T02:58:49.8640440Z [       OK ] HashStoreTest.testStressStore (0 ms)
2022-11-23T02:58:49.8641448Z [ RUN      ] HashStoreTest.testStressStoreWithPrefix
2022-11-23T02:58:49.8647242Z [       OK ] HashStoreTest.testStressStoreWithPrefix (0 ms)
2022-11-23T02:58:49.8647824Z [----------] 4 tests from HashStoreTest (202 ms total)
2022-11-23T02:58:49.8648065Z 
2022-11-23T02:58:49.8648337Z [----------] Global test environment tear-down
2022-11-23T02:58:49.8649995Z [==========] 4 tests from 1 test suite ran. (202 ms total)
2022-11-23T02:58:49.8650436Z [  PASSED  ] 4 tests.
2022-11-23T02:58:49.9320336Z + /opt/conda/lib/python3.10/site-packages/torch/bin/TCPStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml
2022-11-23T02:58:50.2647102Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-11-23T02:58:50.2648013Z [==========] Running 11 tests from 1 test suite.
2022-11-23T02:58:50.2648486Z [----------] Global test environment set-up.
2022-11-23T02:58:50.2648951Z [----------] 11 tests from TCPStoreTest
2022-11-23T02:58:50.2649409Z [ RUN      ] TCPStoreTest.testHelper
2022-11-23T02:58:51.2410230Z [       OK ] TCPStoreTest.testHelper (976 ms)
2022-11-23T02:58:51.2410768Z [ RUN      ] TCPStoreTest.testHelperPrefix
2022-11-23T02:58:52.2011323Z [       OK ] TCPStoreTest.testHelperPrefix (960 ms)
2022-11-23T02:58:52.2011905Z [ RUN      ] TCPStoreTest.testWatchKeyCallback
2022-11-23T02:58:52.2149651Z [       OK ] TCPStoreTest.testWatchKeyCallback (13 ms)
2022-11-23T02:58:52.2150472Z [ RUN      ] TCPStoreTest.testWatchKeyCallbackWithPrefix
2022-11-23T02:58:52.2289676Z [       OK ] TCPStoreTest.testWatchKeyCallbackWithPrefix (13 ms)
2022-11-23T02:58:52.2290321Z [ RUN      ] TCPStoreTest.testKeyEmptyUpdate
2022-11-23T02:58:52.4372911Z [       OK ] TCPStoreTest.testKeyEmptyUpdate (208 ms)
2022-11-23T02:58:52.4373622Z [ RUN      ] TCPStoreTest.testKeyUpdate
2022-11-23T02:58:52.4379067Z [       OK ] TCPStoreTest.testKeyUpdate (0 ms)
2022-11-23T02:58:52.4379734Z [ RUN      ] TCPStoreTest.testKeyCreate
2022-11-23T02:58:52.4384119Z [       OK ] TCPStoreTest.testKeyCreate (0 ms)
2022-11-23T02:58:52.4384774Z [ RUN      ] TCPStoreTest.testKeyAdd
2022-11-23T02:58:52.4388774Z [       OK ] TCPStoreTest.testKeyAdd (0 ms)
2022-11-23T02:58:52.4389630Z [ RUN      ] TCPStoreTest.testKeyDelete
2022-11-23T02:58:52.6451783Z [       OK ] TCPStoreTest.testKeyDelete (206 ms)
2022-11-23T02:58:52.6452458Z [ RUN      ] TCPStoreTest.testCleanShutdown
2022-11-23T02:58:52.6459241Z [       OK ] TCPStoreTest.testCleanShutdown (0 ms)
2022-11-23T02:58:52.6459932Z [ RUN      ] TCPStoreTest.testMultiTenantStores
2022-11-23T02:58:52.6471982Z [       OK ] TCPStoreTest.testMultiTenantStores (1 ms)
2022-11-23T02:58:52.6473090Z [----------] 11 tests from TCPStoreTest (2382 ms total)
2022-11-23T02:58:52.6473430Z 
2022-11-23T02:58:52.6473716Z [----------] Global test environment tear-down
2022-11-23T02:58:52.6475044Z [==========] 11 tests from 1 test suite ran. (2382 ms total)
2022-11-23T02:58:52.6475906Z [  PASSED  ] 11 tests.
2022-11-23T02:58:52.7189173Z ++ command -v mpiexec
2022-11-23T02:58:52.7191605Z + MPIEXEC=/usr/bin/mpiexec
2022-11-23T02:58:52.7192361Z + [[ -n /usr/bin/mpiexec ]]
2022-11-23T02:58:52.7193040Z + [[ -z true ]]
2022-11-23T02:58:52.7193844Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml
2022-11-23T02:58:53.0508696Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-11-23T02:58:53.0509970Z [==========] Running 12 tests from 1 test suite.
2022-11-23T02:58:53.0510465Z [----------] Global test environment set-up.
2022-11-23T02:58:53.0510953Z [----------] 12 tests from ProcessGroupGlooTest
2022-11-23T02:58:53.0511510Z [ RUN      ] ProcessGroupGlooTest.testSIGSTOPException
2022-11-23T02:58:54.0998601Z [       OK ] ProcessGroupGlooTest.testSIGSTOPException (1048 ms)
2022-11-23T02:58:54.0999558Z [ RUN      ] ProcessGroupGlooTest.testSIGKILLException
2022-11-23T02:58:54.1229611Z [       OK ] ProcessGroupGlooTest.testSIGKILLException (23 ms)
2022-11-23T02:58:54.1230618Z [ RUN      ] ProcessGroupGlooTest.testAllReduceCPU
2022-11-23T02:58:54.4105669Z [       OK ] ProcessGroupGlooTest.testAllReduceCPU (287 ms)
2022-11-23T02:58:54.4106662Z [ RUN      ] ProcessGroupGlooTest.testBroadcastCPU
2022-11-23T02:58:54.4528655Z [       OK ] ProcessGroupGlooTest.testBroadcastCPU (42 ms)
2022-11-23T02:58:54.4529799Z [ RUN      ] ProcessGroupGlooTest.testAllToAllCPU
2022-11-23T02:58:54.6058885Z [       OK ] ProcessGroupGlooTest.testAllToAllCPU (152 ms)
2022-11-23T02:58:54.6059898Z [ RUN      ] ProcessGroupGlooTest.testBarrier
2022-11-23T02:58:54.6572210Z [       OK ] ProcessGroupGlooTest.testBarrier (51 ms)
2022-11-23T02:58:54.6573243Z [ RUN      ] ProcessGroupGlooTest.testMonitoredBarrier
2022-11-23T02:58:55.6855908Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 1000 ms
2022-11-23T02:58:55.7058280Z [       OK ] ProcessGroupGlooTest.testMonitoredBarrier (1048 ms)
2022-11-23T02:58:55.7058901Z [ RUN      ] ProcessGroupGlooTest.testSequenceNumInit
2022-11-23T02:58:55.8479231Z [       OK ] ProcessGroupGlooTest.testSequenceNumInit (142 ms)
2022-11-23T02:58:55.8479798Z [ RUN      ] ProcessGroupGlooTest.testSend
2022-11-23T02:58:55.8894169Z [       OK ] ProcessGroupGlooTest.testSend (41 ms)
2022-11-23T02:58:55.8895061Z [ RUN      ] ProcessGroupGlooTest.testRecv
2022-11-23T02:58:55.9409030Z [       OK ] ProcessGroupGlooTest.testRecv (51 ms)
2022-11-23T02:58:55.9409584Z [ RUN      ] ProcessGroupGlooTest.testStoreSetGet
2022-11-23T02:58:55.9918890Z [       OK ] ProcessGroupGlooTest.testStoreSetGet (50 ms)
2022-11-23T02:58:55.9919473Z [ RUN      ] ProcessGroupGlooTest.testWaitDelay
2022-11-23T02:58:56.1335653Z [       OK ] ProcessGroupGlooTest.testWaitDelay (141 ms)
2022-11-23T02:58:56.1336380Z [----------] 12 tests from ProcessGroupGlooTest (3082 ms total)
2022-11-23T02:58:56.1336684Z 
2022-11-23T02:58:56.1336961Z [----------] Global test environment tear-down
2022-11-23T02:58:56.1339086Z [==========] 12 tests from 1 test suite ran. (3082 ms total)
2022-11-23T02:58:56.1339514Z [  PASSED  ] 12 tests.
2022-11-23T02:58:56.2155544Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml
2022-11-23T02:58:56.6435043Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-11-23T02:58:56.6435743Z [==========] Running 11 tests from 1 test suite.
2022-11-23T02:58:56.6436358Z [----------] Global test environment set-up.
2022-11-23T02:58:56.6436860Z [----------] 11 tests from ProcessGroupNCCLTest
2022-11-23T02:58:56.6437370Z [ RUN      ] ProcessGroupNCCLTest.testAllreduce
2022-11-23T02:59:01.1153599Z [       OK ] ProcessGroupNCCLTest.testAllreduce (4471 ms)
2022-11-23T02:59:01.1154201Z [ RUN      ] ProcessGroupNCCLTest.testBroadcast
2022-11-23T02:59:04.6129578Z [       OK ] ProcessGroupNCCLTest.testBroadcast (3497 ms)
2022-11-23T02:59:04.6130169Z [ RUN      ] ProcessGroupNCCLTest.testReduce
2022-11-23T02:59:08.1106780Z [       OK ] ProcessGroupNCCLTest.testReduce (3497 ms)
2022-11-23T02:59:08.1107363Z [ RUN      ] ProcessGroupNCCLTest.testAllgather
2022-11-23T02:59:09.8965077Z [       OK ] ProcessGroupNCCLTest.testAllgather (1785 ms)
2022-11-23T02:59:09.8965949Z [ RUN      ] ProcessGroupNCCLTest.testAllgatherBase
2022-11-23T02:59:11.6656021Z [       OK ] ProcessGroupNCCLTest.testAllgatherBase (1768 ms)
2022-11-23T02:59:11.6656657Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatter
2022-11-23T02:59:13.4466838Z [       OK ] ProcessGroupNCCLTest.testReduceScatter (1781 ms)
2022-11-23T02:59:13.4467479Z [ RUN      ] ProcessGroupNCCLTest.testSequenceNumInit
2022-11-23T02:59:13.5565473Z [       OK ] ProcessGroupNCCLTest.testSequenceNumInit (109 ms)
2022-11-23T02:59:13.5566764Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout
2022-11-23T02:59:16.5612786Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout (3004 ms)
2022-11-23T02:59:16.5614534Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException
2022-11-23T02:59:19.5625334Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException (3001 ms)
2022-11-23T02:59:19.5626311Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatterBase
2022-11-23T02:59:21.6919596Z [       OK ] ProcessGroupNCCLTest.testReduceScatterBase (2129 ms)
2022-11-23T02:59:21.6920279Z [ RUN      ] ProcessGroupNCCLTest.testBackendName
2022-11-23T02:59:21.7340678Z [       OK ] ProcessGroupNCCLTest.testBackendName (42 ms)
2022-11-23T02:59:21.7341769Z [----------] 11 tests from ProcessGroupNCCLTest (25090 ms total)
2022-11-23T02:59:21.7343574Z 
2022-11-23T02:59:21.7344547Z [----------] Global test environment tear-down
2022-11-23T02:59:21.7345279Z [==========] 11 tests from 1 test suite ran. (25090 ms total)
2022-11-23T02:59:21.7345745Z [  PASSED  ] 11 tests.
2022-11-23T02:59:21.9514274Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml
2022-11-23T02:59:22.3796315Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-11-23T02:59:22.3797053Z [==========] Running 3 tests from 1 test suite.
2022-11-23T02:59:22.3797658Z [----------] Global test environment set-up.
2022-11-23T02:59:22.3798189Z [----------] 3 tests from ProcessGroupNCCLErrorsTest
2022-11-23T02:59:22.3799054Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking
2022-11-23T02:59:24.8504448Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking (2470 ms)
2022-11-23T02:59:24.8505221Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking
2022-11-23T02:59:27.9520165Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking (3101 ms)
2022-11-23T02:59:27.9520962Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking
2022-11-23T02:59:28.0509470Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking (98 ms)
2022-11-23T02:59:28.0510188Z [----------] 3 tests from ProcessGroupNCCLErrorsTest (5671 ms total)
2022-11-23T02:59:28.0510496Z 
2022-11-23T02:59:28.0510769Z [----------] Global test environment tear-down
2022-11-23T02:59:28.0513645Z [==========] 3 tests from 1 test suite ran. (5671 ms total)
2022-11-23T02:59:28.0514118Z [  PASSED  ] 3 tests.
2022-11-23T02:59:28.2702703Z + [[ 1 == 1 ]]
2022-11-23T02:59:28.2703111Z + test_rpc
2022-11-23T02:59:28.2703695Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *rocm* ]]
2022-11-23T02:59:28.2704155Z + echo 'Testing RPC C++ tests'
2022-11-23T02:59:28.2704503Z Testing RPC C++ tests
2022-11-23T02:59:28.2705969Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T02:59:28.2720617Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T02:59:28.2733110Z + ln -sf '/opt/conda/lib/python3.10/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T02:59:28.2743133Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc
2022-11-23T02:59:28.2743916Z + mkdir -p test/test-reports/cpp-rpc/test_rpc
2022-11-23T02:59:28.2756584Z + /opt/conda/lib/python3.10/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml
2022-11-23T02:59:28.7395770Z [==========] Running 8 tests from 3 test suites.
2022-11-23T02:59:28.7396363Z [----------] Global test environment set-up.
2022-11-23T02:59:28.7396949Z [----------] 4 tests from WireSerialize
2022-11-23T02:59:28.7397389Z [ RUN      ] WireSerialize.Base
2022-11-23T02:59:28.7409849Z [       OK ] WireSerialize.Base (1 ms)
2022-11-23T02:59:28.7410367Z [ RUN      ] WireSerialize.RecopySparseTensors
2022-11-23T02:59:28.7515720Z [       OK ] WireSerialize.RecopySparseTensors (10 ms)
2022-11-23T02:59:28.7516268Z [ RUN      ] WireSerialize.CloneSparseTensors
2022-11-23T02:59:28.7614995Z [       OK ] WireSerialize.CloneSparseTensors (9 ms)
2022-11-23T02:59:28.7615480Z [ RUN      ] WireSerialize.Errors
2022-11-23T02:59:28.7642195Z [       OK ] WireSerialize.Errors (2 ms)
2022-11-23T02:59:28.7642685Z [----------] 4 tests from WireSerialize (24 ms total)
2022-11-23T02:59:28.7642946Z 
2022-11-23T02:59:28.7643201Z [----------] 1 test from TestE2ETensorPipe
2022-11-23T02:59:28.7643757Z [ RUN      ] TestE2ETensorPipe.TestTrainingLoop
2022-11-23T02:59:29.4628077Z [W tensorpipe_agent.cpp:725] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T02:59:29.4645300Z [       OK ] TestE2ETensorPipe.TestTrainingLoop (700 ms)
2022-11-23T02:59:29.4646565Z [----------] 1 test from TestE2ETensorPipe (700 ms total)
2022-11-23T02:59:29.4646919Z 
2022-11-23T02:59:29.4647221Z [----------] 3 tests from TensorpipeSerialize
2022-11-23T02:59:29.4647700Z [ RUN      ] TensorpipeSerialize.Base
2022-11-23T02:59:29.4648178Z [       OK ] TensorpipeSerialize.Base (0 ms)
2022-11-23T02:59:29.4648715Z [ RUN      ] TensorpipeSerialize.RecopySparseTensors
2022-11-23T02:59:29.4748869Z [       OK ] TensorpipeSerialize.RecopySparseTensors (10 ms)
2022-11-23T02:59:29.4749876Z [ RUN      ] TensorpipeSerialize.NoDeleterTensors
2022-11-23T02:59:29.4750834Z [       OK ] TensorpipeSerialize.NoDeleterTensors (0 ms)
2022-11-23T02:59:29.4751782Z [----------] 3 tests from TensorpipeSerialize (10 ms total)
2022-11-23T02:59:29.4752301Z 
2022-11-23T02:59:29.4752759Z [----------] Global test environment tear-down
2022-11-23T02:59:29.4753630Z [==========] 8 tests from 3 test suites ran. (735 ms total)
2022-11-23T02:59:29.4754324Z [  PASSED  ] 8 tests.
2022-11-23T02:59:29.4754652Z 
2022-11-23T02:59:29.4754962Z   YOU HAVE 1 DISABLED TEST
2022-11-23T02:59:29.4755310Z 
2022-11-23T02:59:29.5789015Z 
2022-11-23T02:59:29.5825354Z Prepare all required actions
2022-11-23T02:59:29.5825884Z Getting action download info
2022-11-23T02:59:29.7279223Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2022-11-23T02:59:29.8699100Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-11-23T02:59:29.8699447Z with:
2022-11-23T02:59:29.8699962Z   github-token: ***
2022-11-23T02:59:29.8700271Z env:
2022-11-23T02:59:29.8700550Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:29.8700881Z   GPU_FLAG: --gpus all
2022-11-23T02:59:29.8701327Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:29.8701757Z ##[endgroup]
2022-11-23T02:59:29.8736462Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2022-11-23T02:59:29.8736854Z with:
2022-11-23T02:59:29.8737150Z   shell: bash
2022-11-23T02:59:29.8737434Z   timeout_minutes: 10
2022-11-23T02:59:29.8737761Z   max_attempts: 5
2022-11-23T02:59:29.8738106Z   retry_wait_seconds: 30
2022-11-23T02:59:29.8738812Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"

2022-11-23T02:59:29.8739386Z   polling_interval_seconds: 1
2022-11-23T02:59:29.8739719Z   warning_on_retry: true
2022-11-23T02:59:29.8740042Z   continue_on_error: false
2022-11-23T02:59:29.8740334Z env:
2022-11-23T02:59:29.8740625Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:29.8740953Z   GPU_FLAG: --gpus all
2022-11-23T02:59:29.8741389Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:29.8741986Z   GITHUB_TOKEN: ***
2022-11-23T02:59:29.8742304Z ##[endgroup]
2022-11-23T02:59:29.9412518Z + python3 -m pip install requests==2.26.0
2022-11-23T02:59:30.2430137Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T02:59:30.2656955Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2022-11-23T02:59:30.2841724Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.9.24)
2022-11-23T02:59:30.2854382Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.12)
2022-11-23T02:59:30.3077952Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2022-11-23T02:59:30.3104364Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4)
2022-11-23T02:59:30.5697417Z ++ python3 .github/scripts/get_workflow_job_id.py 3528293554 i-0274d02e8f37638bd
2022-11-23T02:59:32.8271524Z + GHA_WORKFLOW_JOB_ID=9655199096
2022-11-23T02:59:32.8273923Z + echo job-id=9655199096
2022-11-23T02:59:32.9404252Z Command completed after 1 attempt(s).
2022-11-23T02:59:32.9538658Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-11-23T02:59:32.9539088Z kill "$MONITOR_SCRIPT_PID"
2022-11-23T02:59:32.9552334Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:59:32.9552713Z env:
2022-11-23T02:59:32.9553032Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:32.9553365Z   GPU_FLAG: --gpus all
2022-11-23T02:59:32.9553835Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:32.9554313Z   MONITOR_SCRIPT_PID: 81816
2022-11-23T02:59:32.9554634Z ##[endgroup]
2022-11-23T02:59:32.9674130Z Prepare all required actions
2022-11-23T02:59:32.9674544Z Getting action download info
2022-11-23T02:59:33.1392038Z Download action repository 'actions/upload-artifact@v3' (SHA:83fd05a356d7e2593de66fc9913b3002723633cb)
2022-11-23T02:59:34.3117228Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-11-23T02:59:34.3117604Z with:
2022-11-23T02:59:34.3118023Z   file-suffix: test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655199096
2022-11-23T02:59:34.3118443Z env:
2022-11-23T02:59:34.3118723Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:34.3119053Z   GPU_FLAG: --gpus all
2022-11-23T02:59:34.3119507Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:34.3119932Z ##[endgroup]
2022-11-23T02:59:34.3154597Z ##[group]Run # Remove any previous test jsons if they exist
2022-11-23T02:59:34.3155021Z # Remove any previous test jsons if they exist
2022-11-23T02:59:34.3155391Z rm -f test-jsons-*.zip
2022-11-23T02:59:34.3155821Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-11-23T02:59:34.3168626Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:59:34.3168996Z env:
2022-11-23T02:59:34.3169294Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:34.3169605Z   GPU_FLAG: --gpus all
2022-11-23T02:59:34.3170056Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:34.3170745Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655199096
2022-11-23T02:59:34.3171152Z ##[endgroup]
2022-11-23T02:59:34.3293282Z   adding: test/allowlist_for_publicAPI.json (deflated 79%)
2022-11-23T02:59:34.3328606Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-11-23T02:59:34.3336124Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2022-11-23T02:59:34.3337180Z   adding: test/.pytorch-slow-tests.json (deflated 73%)
2022-11-23T02:59:34.3347979Z   adding: test/.pytorch-disabled-tests.json (deflated 86%)
2022-11-23T02:59:34.3373703Z ##[group]Run # Remove any previous test reports if they exist
2022-11-23T02:59:34.3374162Z # Remove any previous test reports if they exist
2022-11-23T02:59:34.3374547Z rm -f test-reports-*.zip
2022-11-23T02:59:34.3374969Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2022-11-23T02:59:34.3386747Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:59:34.3387106Z env:
2022-11-23T02:59:34.3387405Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:34.3387723Z   GPU_FLAG: --gpus all
2022-11-23T02:59:34.3388171Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:34.3388734Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655199096
2022-11-23T02:59:34.3389139Z ##[endgroup]
2022-11-23T02:59:34.3540632Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011912.xml (deflated 41%)
2022-11-23T02:59:34.3541651Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011918.xml (deflated 42%)
2022-11-23T02:59:34.3542615Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011924.xml (deflated 44%)
2022-11-23T02:59:34.3543576Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011926.xml (deflated 44%)
2022-11-23T02:59:34.3544532Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011928.xml (deflated 44%)
2022-11-23T02:59:34.3545490Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011931.xml (deflated 45%)
2022-11-23T02:59:34.3546590Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011935.xml (deflated 41%)
2022-11-23T02:59:34.3547568Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011941.xml (deflated 41%)
2022-11-23T02:59:34.3548506Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011947.xml (deflated 44%)
2022-11-23T02:59:34.3549479Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011949.xml (deflated 44%)
2022-11-23T02:59:34.3550435Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011951.xml (deflated 44%)
2022-11-23T02:59:34.3551376Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123011954.xml (deflated 45%)
2022-11-23T02:59:34.3552291Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012007.xml (deflated 41%)
2022-11-23T02:59:34.3553138Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012016.xml (deflated 42%)
2022-11-23T02:59:34.3554004Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012018.xml (deflated 42%)
2022-11-23T02:59:34.3554977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012025.xml (deflated 43%)
2022-11-23T02:59:34.3555831Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012029.xml (deflated 42%)
2022-11-23T02:59:34.3556670Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012036.xml (deflated 41%)
2022-11-23T02:59:34.3557537Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012043.xml (deflated 40%)
2022-11-23T02:59:34.3558408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012051.xml (deflated 40%)
2022-11-23T02:59:34.3559263Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012059.xml (deflated 41%)
2022-11-23T02:59:34.3560126Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012106.xml (deflated 40%)
2022-11-23T02:59:34.3560969Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012114.xml (deflated 40%)
2022-11-23T02:59:34.3561815Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012123.xml (deflated 41%)
2022-11-23T02:59:34.3562675Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012131.xml (deflated 40%)
2022-11-23T02:59:34.3563523Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012138.xml (deflated 42%)
2022-11-23T02:59:34.3564357Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012143.xml (deflated 41%)
2022-11-23T02:59:34.3565223Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012149.xml (deflated 42%)
2022-11-23T02:59:34.3566339Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012154.xml (deflated 42%)
2022-11-23T02:59:34.3567197Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012201.xml (deflated 42%)
2022-11-23T02:59:34.3568119Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012204.xml (deflated 46%)
2022-11-23T02:59:34.3568981Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012210.xml (deflated 47%)
2022-11-23T02:59:34.3569833Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012217.xml (deflated 48%)
2022-11-23T02:59:34.3570693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012224.xml (deflated 46%)
2022-11-23T02:59:34.3571522Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012231.xml (deflated 40%)
2022-11-23T02:59:34.3572379Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012238.xml (deflated 41%)
2022-11-23T02:59:34.3573259Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012245.xml (deflated 41%)
2022-11-23T02:59:34.3574109Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012251.xml (deflated 42%)
2022-11-23T02:59:34.3574942Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012258.xml (deflated 41%)
2022-11-23T02:59:34.3575789Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012305.xml (deflated 41%)
2022-11-23T02:59:34.3576726Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012312.xml (deflated 40%)
2022-11-23T02:59:34.3577576Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012319.xml (deflated 41%)
2022-11-23T02:59:34.3578403Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012321.xml (deflated 41%)
2022-11-23T02:59:34.3579247Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012323.xml (deflated 41%)
2022-11-23T02:59:34.3580087Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012330.xml (deflated 42%)
2022-11-23T02:59:34.3580938Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012337.xml (deflated 42%)
2022-11-23T02:59:34.3581790Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012339.xml (deflated 42%)
2022-11-23T02:59:34.3582625Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012342.xml (deflated 42%)
2022-11-23T02:59:34.3583469Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012344.xml (deflated 42%)
2022-11-23T02:59:34.3584317Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012346.xml (deflated 40%)
2022-11-23T02:59:34.3585157Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012353.xml (deflated 40%)
2022-11-23T02:59:34.3585994Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012400.xml (deflated 42%)
2022-11-23T02:59:34.3586842Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012402.xml (deflated 40%)
2022-11-23T02:59:34.3587684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012409.xml (deflated 41%)
2022-11-23T02:59:34.3588532Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012416.xml (deflated 40%)
2022-11-23T02:59:34.3589414Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012423.xml (deflated 40%)
2022-11-23T02:59:34.3590269Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012430.xml (deflated 42%)
2022-11-23T02:59:34.3591116Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012437.xml (deflated 42%)
2022-11-23T02:59:34.3591979Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012443.xml (deflated 42%)
2022-11-23T02:59:34.3592803Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012450.xml (deflated 42%)
2022-11-23T02:59:34.3593646Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012457.xml (deflated 40%)
2022-11-23T02:59:34.3594495Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012504.xml (deflated 41%)
2022-11-23T02:59:34.3595340Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012511.xml (deflated 41%)
2022-11-23T02:59:34.3596173Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012517.xml (deflated 41%)
2022-11-23T02:59:34.3597094Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012524.xml (deflated 41%)
2022-11-23T02:59:34.3597936Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012531.xml (deflated 41%)
2022-11-23T02:59:34.3598776Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012538.xml (deflated 40%)
2022-11-23T02:59:34.3599668Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012545.xml (deflated 40%)
2022-11-23T02:59:34.3600498Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012552.xml (deflated 41%)
2022-11-23T02:59:34.3601337Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012558.xml (deflated 40%)
2022-11-23T02:59:34.3602188Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012605.xml (deflated 42%)
2022-11-23T02:59:34.3603036Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012612.xml (deflated 42%)
2022-11-23T02:59:34.3603870Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012619.xml (deflated 42%)
2022-11-23T02:59:34.3604729Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012626.xml (deflated 42%)
2022-11-23T02:59:34.3605571Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012632.xml (deflated 40%)
2022-11-23T02:59:34.3606607Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012639.xml (deflated 41%)
2022-11-23T02:59:34.3607443Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012646.xml (deflated 40%)
2022-11-23T02:59:34.3608289Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012654.xml (deflated 41%)
2022-11-23T02:59:34.3609139Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012702.xml (deflated 40%)
2022-11-23T02:59:34.3609987Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012709.xml (deflated 41%)
2022-11-23T02:59:34.3610911Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012717.xml (deflated 41%)
2022-11-23T02:59:34.3611772Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012723.xml (deflated 40%)
2022-11-23T02:59:34.3612617Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012730.xml (deflated 40%)
2022-11-23T02:59:34.3613468Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012737.xml (deflated 40%)
2022-11-23T02:59:34.3614299Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012746.xml (deflated 40%)
2022-11-23T02:59:34.3615149Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012754.xml (deflated 41%)
2022-11-23T02:59:34.3615999Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012803.xml (deflated 42%)
2022-11-23T02:59:34.3616847Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012805.xml (deflated 42%)
2022-11-23T02:59:34.3617675Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012808.xml (deflated 42%)
2022-11-23T02:59:34.3618613Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012810.xml (deflated 42%)
2022-11-23T02:59:34.3619456Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012812.xml (deflated 43%)
2022-11-23T02:59:34.3620302Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012815.xml (deflated 42%)
2022-11-23T02:59:34.3621131Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012817.xml (deflated 42%)
2022-11-23T02:59:34.3621980Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012820.xml (deflated 42%)
2022-11-23T02:59:34.3622818Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012822.xml (deflated 42%)
2022-11-23T02:59:34.3623662Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012824.xml (deflated 42%)
2022-11-23T02:59:34.3624493Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012827.xml (deflated 43%)
2022-11-23T02:59:34.3625343Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012829.xml (deflated 42%)
2022-11-23T02:59:34.3626195Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012832.xml (deflated 42%)
2022-11-23T02:59:34.3627035Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012834.xml (deflated 42%)
2022-11-23T02:59:34.3627860Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012836.xml (deflated 42%)
2022-11-23T02:59:34.3628717Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012839.xml (deflated 43%)
2022-11-23T02:59:34.3629600Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012841.xml (deflated 42%)
2022-11-23T02:59:34.3630440Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012844.xml (deflated 42%)
2022-11-23T02:59:34.3631331Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012846.xml (deflated 42%)
2022-11-23T02:59:34.3632179Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012848.xml (deflated 42%)
2022-11-23T02:59:34.3633021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012851.xml (deflated 42%)
2022-11-23T02:59:34.3633875Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012853.xml (deflated 42%)
2022-11-23T02:59:34.3634692Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012856.xml (deflated 42%)
2022-11-23T02:59:34.3635534Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012858.xml (deflated 43%)
2022-11-23T02:59:34.3636384Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012900.xml (deflated 41%)
2022-11-23T02:59:34.3637229Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012909.xml (deflated 42%)
2022-11-23T02:59:34.3638056Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012915.xml (deflated 42%)
2022-11-23T02:59:34.3638917Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012918.xml (deflated 40%)
2022-11-23T02:59:34.3639817Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012925.xml (deflated 41%)
2022-11-23T02:59:34.3640659Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012934.xml (deflated 40%)
2022-11-23T02:59:34.3641507Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012941.xml (deflated 42%)
2022-11-23T02:59:34.3642345Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012948.xml (deflated 42%)
2022-11-23T02:59:34.3643188Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012955.xml (deflated 42%)
2022-11-23T02:59:34.3644031Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013002.xml (deflated 40%)
2022-11-23T02:59:34.3644932Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013010.xml (deflated 42%)
2022-11-23T02:59:34.3645942Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013012.xml (deflated 42%)
2022-11-23T02:59:34.3646785Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013019.xml (deflated 41%)
2022-11-23T02:59:34.3647629Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013026.xml (deflated 40%)
2022-11-23T02:59:34.3648485Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013032.xml (deflated 42%)
2022-11-23T02:59:34.3649326Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013035.xml (deflated 41%)
2022-11-23T02:59:34.3650159Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013037.xml (deflated 41%)
2022-11-23T02:59:34.3651006Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013040.xml (deflated 41%)
2022-11-23T02:59:34.3651849Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013042.xml (deflated 41%)
2022-11-23T02:59:34.3652771Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013044.xml (deflated 41%)
2022-11-23T02:59:34.3653612Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013047.xml (deflated 41%)
2022-11-23T02:59:34.3654453Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013049.xml (deflated 41%)
2022-11-23T02:59:34.3655312Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013052.xml (deflated 41%)
2022-11-23T02:59:34.3656155Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013058.xml (deflated 42%)
2022-11-23T02:59:34.3656974Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013103.xml (deflated 40%)
2022-11-23T02:59:34.3657838Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013109.xml (deflated 42%)
2022-11-23T02:59:34.3658678Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013116.xml (deflated 41%)
2022-11-23T02:59:34.3659519Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013124.xml (deflated 42%)
2022-11-23T02:59:34.3660450Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013128.xml (deflated 42%)
2022-11-23T02:59:34.3661296Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013132.xml (deflated 42%)
2022-11-23T02:59:34.3662138Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013136.xml (deflated 40%)
2022-11-23T02:59:34.3662984Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013144.xml (deflated 40%)
2022-11-23T02:59:34.3663817Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013152.xml (deflated 42%)
2022-11-23T02:59:34.3664660Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013156.xml (deflated 42%)
2022-11-23T02:59:34.3665503Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013200.xml (deflated 40%)
2022-11-23T02:59:34.3666346Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013208.xml (deflated 40%)
2022-11-23T02:59:34.3667166Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013216.xml (deflated 41%)
2022-11-23T02:59:34.3668019Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013224.xml (deflated 40%)
2022-11-23T02:59:34.3668868Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013232.xml (deflated 42%)
2022-11-23T02:59:34.3669704Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013236.xml (deflated 41%)
2022-11-23T02:59:34.3670536Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013243.xml (deflated 42%)
2022-11-23T02:59:34.3671388Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013247.xml (deflated 41%)
2022-11-23T02:59:34.3672235Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013255.xml (deflated 42%)
2022-11-23T02:59:34.3673074Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013259.xml (deflated 42%)
2022-11-23T02:59:34.3673949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013303.xml (deflated 40%)
2022-11-23T02:59:34.3674800Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013312.xml (deflated 40%)
2022-11-23T02:59:34.3675652Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013321.xml (deflated 42%)
2022-11-23T02:59:34.3676499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013325.xml (deflated 41%)
2022-11-23T02:59:34.3677331Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013333.xml (deflated 41%)
2022-11-23T02:59:34.3678174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013342.xml (deflated 41%)
2022-11-23T02:59:34.3679023Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013350.xml (deflated 41%)
2022-11-23T02:59:34.3679873Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013358.xml (deflated 40%)
2022-11-23T02:59:34.3680713Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013407.xml (deflated 40%)
2022-11-23T02:59:34.3681609Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013415.xml (deflated 41%)
2022-11-23T02:59:34.3682456Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013423.xml (deflated 40%)
2022-11-23T02:59:34.3683306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013431.xml (deflated 40%)
2022-11-23T02:59:34.3684147Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013440.xml (deflated 41%)
2022-11-23T02:59:34.3684980Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013448.xml (deflated 41%)
2022-11-23T02:59:34.3686097Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013456.xml (deflated 41%)
2022-11-23T02:59:34.3686955Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013505.xml (deflated 40%)
2022-11-23T02:59:34.3687800Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013513.xml (deflated 42%)
2022-11-23T02:59:34.3688622Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013517.xml (deflated 41%)
2022-11-23T02:59:34.3689472Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013525.xml (deflated 41%)
2022-11-23T02:59:34.3690323Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013534.xml (deflated 40%)
2022-11-23T02:59:34.3691163Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013540.xml (deflated 40%)
2022-11-23T02:59:34.3691999Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013549.xml (deflated 40%)
2022-11-23T02:59:34.3692848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013556.xml (deflated 40%)
2022-11-23T02:59:34.3693698Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013614.xml (deflated 41%)
2022-11-23T02:59:34.3694617Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013622.xml (deflated 41%)
2022-11-23T02:59:34.3695464Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013631.xml (deflated 41%)
2022-11-23T02:59:34.3696309Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013639.xml (deflated 40%)
2022-11-23T02:59:34.3697156Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013647.xml (deflated 42%)
2022-11-23T02:59:34.3698001Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013651.xml (deflated 43%)
2022-11-23T02:59:34.3698827Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013655.xml (deflated 42%)
2022-11-23T02:59:34.3699683Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013704.xml (deflated 40%)
2022-11-23T02:59:34.3700527Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013712.xml (deflated 42%)
2022-11-23T02:59:34.3701363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013716.xml (deflated 41%)
2022-11-23T02:59:34.3702189Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013724.xml (deflated 42%)
2022-11-23T02:59:34.3703127Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013728.xml (deflated 41%)
2022-11-23T02:59:34.3703969Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013737.xml (deflated 41%)
2022-11-23T02:59:34.3704816Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013744.xml (deflated 40%)
2022-11-23T02:59:34.3705642Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013752.xml (deflated 42%)
2022-11-23T02:59:34.3706491Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013756.xml (deflated 42%)
2022-11-23T02:59:34.3707340Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013800.xml (deflated 42%)
2022-11-23T02:59:34.3708192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013805.xml (deflated 40%)
2022-11-23T02:59:34.3709010Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013813.xml (deflated 41%)
2022-11-23T02:59:34.3709856Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013820.xml (deflated 40%)
2022-11-23T02:59:34.3710709Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013827.xml (deflated 41%)
2022-11-23T02:59:34.3711551Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013834.xml (deflated 42%)
2022-11-23T02:59:34.3712374Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013838.xml (deflated 42%)
2022-11-23T02:59:34.3713231Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013842.xml (deflated 40%)
2022-11-23T02:59:34.3714064Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013849.xml (deflated 40%)
2022-11-23T02:59:34.3714912Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013856.xml (deflated 40%)
2022-11-23T02:59:34.3715800Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013903.xml (deflated 42%)
2022-11-23T02:59:34.3716652Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013905.xml (deflated 40%)
2022-11-23T02:59:34.3717496Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013912.xml (deflated 42%)
2022-11-23T02:59:34.3718344Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013919.xml (deflated 40%)
2022-11-23T02:59:34.3719177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013926.xml (deflated 42%)
2022-11-23T02:59:34.3720015Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013930.xml (deflated 41%)
2022-11-23T02:59:34.3720951Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013937.xml (deflated 41%)
2022-11-23T02:59:34.3721778Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013944.xml (deflated 40%)
2022-11-23T02:59:34.3722623Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013951.xml (deflated 40%)
2022-11-23T02:59:34.3723531Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013957.xml (deflated 40%)
2022-11-23T02:59:34.3724379Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014004.xml (deflated 40%)
2022-11-23T02:59:34.3725204Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014013.xml (deflated 41%)
2022-11-23T02:59:34.3726236Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014019.xml (deflated 41%)
2022-11-23T02:59:34.3727083Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014026.xml (deflated 41%)
2022-11-23T02:59:34.3727927Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014033.xml (deflated 41%)
2022-11-23T02:59:34.3728762Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014040.xml (deflated 41%)
2022-11-23T02:59:34.3729653Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014104.xml (deflated 41%)
2022-11-23T02:59:34.3730490Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014129.xml (deflated 42%)
2022-11-23T02:59:34.3731337Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014135.xml (deflated 40%)
2022-11-23T02:59:34.3732173Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014144.xml (deflated 40%)
2022-11-23T02:59:34.3733018Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014151.xml (deflated 40%)
2022-11-23T02:59:34.3733862Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014158.xml (deflated 41%)
2022-11-23T02:59:34.3734710Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014205.xml (deflated 42%)
2022-11-23T02:59:34.3735539Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014207.xml (deflated 42%)
2022-11-23T02:59:34.3736390Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014209.xml (deflated 42%)
2022-11-23T02:59:34.3737302Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014212.xml (deflated 42%)
2022-11-23T02:59:34.3738158Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014214.xml (deflated 42%)
2022-11-23T02:59:34.3738983Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014217.xml (deflated 42%)
2022-11-23T02:59:34.3739839Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014219.xml (deflated 42%)
2022-11-23T02:59:34.3740686Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014221.xml (deflated 42%)
2022-11-23T02:59:34.3741527Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014224.xml (deflated 40%)
2022-11-23T02:59:34.3742358Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014230.xml (deflated 40%)
2022-11-23T02:59:34.3743216Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014237.xml (deflated 41%)
2022-11-23T02:59:34.3744059Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014240.xml (deflated 42%)
2022-11-23T02:59:34.3744977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014242.xml (deflated 42%)
2022-11-23T02:59:34.3745811Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014246.xml (deflated 40%)
2022-11-23T02:59:34.3746655Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014254.xml (deflated 40%)
2022-11-23T02:59:34.3747500Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014303.xml (deflated 40%)
2022-11-23T02:59:34.3748349Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014311.xml (deflated 42%)
2022-11-23T02:59:34.3749163Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014315.xml (deflated 41%)
2022-11-23T02:59:34.3750016Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014319.xml (deflated 40%)
2022-11-23T02:59:34.3750857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014326.xml (deflated 40%)
2022-11-23T02:59:34.3751696Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014332.xml (deflated 42%)
2022-11-23T02:59:34.3752550Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014337.xml (deflated 41%)
2022-11-23T02:59:34.3753378Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014343.xml (deflated 41%)
2022-11-23T02:59:34.3754222Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014350.xml (deflated 41%)
2022-11-23T02:59:34.3755064Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014357.xml (deflated 40%)
2022-11-23T02:59:34.3755900Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014404.xml (deflated 42%)
2022-11-23T02:59:34.3756724Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014411.xml (deflated 42%)
2022-11-23T02:59:34.3757631Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014418.xml (deflated 42%)
2022-11-23T02:59:34.3758477Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014424.xml (deflated 42%)
2022-11-23T02:59:34.3759319Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014431.xml (deflated 41%)
2022-11-23T02:59:34.3760145Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014438.xml (deflated 41%)
2022-11-23T02:59:34.3760992Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014445.xml (deflated 42%)
2022-11-23T02:59:34.3761841Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014447.xml (deflated 41%)
2022-11-23T02:59:34.3762687Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014454.xml (deflated 43%)
2022-11-23T02:59:34.3763520Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014457.xml (deflated 43%)
2022-11-23T02:59:34.3764364Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014459.xml (deflated 40%)
2022-11-23T02:59:34.3765207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014506.xml (deflated 41%)
2022-11-23T02:59:34.3766288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014508.xml (deflated 42%)
2022-11-23T02:59:34.3767126Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014511.xml (deflated 41%)
2022-11-23T02:59:34.3767972Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014518.xml (deflated 41%)
2022-11-23T02:59:34.3768811Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014524.xml (deflated 40%)
2022-11-23T02:59:34.3769654Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014531.xml (deflated 40%)
2022-11-23T02:59:34.3770480Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014538.xml (deflated 41%)
2022-11-23T02:59:34.3771333Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014540.xml (deflated 41%)
2022-11-23T02:59:34.3772167Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014543.xml (deflated 40%)
2022-11-23T02:59:34.3773016Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014550.xml (deflated 42%)
2022-11-23T02:59:34.3773837Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014556.xml (deflated 41%)
2022-11-23T02:59:34.3774686Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014603.xml (deflated 40%)
2022-11-23T02:59:34.3775526Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014610.xml (deflated 40%)
2022-11-23T02:59:34.3776369Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014617.xml (deflated 40%)
2022-11-23T02:59:34.3777210Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014624.xml (deflated 40%)
2022-11-23T02:59:34.3778056Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014630.xml (deflated 41%)
2022-11-23T02:59:34.3778966Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014637.xml (deflated 41%)
2022-11-23T02:59:34.3779822Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014639.xml (deflated 40%)
2022-11-23T02:59:34.3780656Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014642.xml (deflated 41%)
2022-11-23T02:59:34.3781507Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014644.xml (deflated 40%)
2022-11-23T02:59:34.3782355Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014651.xml (deflated 40%)
2022-11-23T02:59:34.3783197Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014658.xml (deflated 40%)
2022-11-23T02:59:34.3784023Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014705.xml (deflated 40%)
2022-11-23T02:59:34.3784878Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014712.xml (deflated 41%)
2022-11-23T02:59:34.3785712Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014719.xml (deflated 40%)
2022-11-23T02:59:34.3786653Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014726.xml (deflated 41%)
2022-11-23T02:59:34.3787486Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014734.xml (deflated 41%)
2022-11-23T02:59:34.3788317Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014741.xml (deflated 40%)
2022-11-23T02:59:34.3789167Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014749.xml (deflated 41%)
2022-11-23T02:59:34.3790005Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014757.xml (deflated 40%)
2022-11-23T02:59:34.3790850Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014810.xml (deflated 40%)
2022-11-23T02:59:34.3791677Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014825.xml (deflated 41%)
2022-11-23T02:59:34.3792522Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014833.xml (deflated 41%)
2022-11-23T02:59:34.3793359Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014835.xml (deflated 41%)
2022-11-23T02:59:34.3794204Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014842.xml (deflated 43%)
2022-11-23T02:59:34.3795035Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014846.xml (deflated 41%)
2022-11-23T02:59:34.3795873Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014853.xml (deflated 41%)
2022-11-23T02:59:34.3796712Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014900.xml (deflated 40%)
2022-11-23T02:59:34.3797560Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014909.xml (deflated 40%)
2022-11-23T02:59:34.3798388Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014916.xml (deflated 40%)
2022-11-23T02:59:34.3799231Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014924.xml (deflated 39%)
2022-11-23T02:59:34.3800126Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014932.xml (deflated 39%)
2022-11-23T02:59:34.3800975Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014940.xml (deflated 40%)
2022-11-23T02:59:34.3801803Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014949.xml (deflated 40%)
2022-11-23T02:59:34.3802656Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014956.xml (deflated 42%)
2022-11-23T02:59:34.3803491Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015000.xml (deflated 42%)
2022-11-23T02:59:34.3804335Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015007.xml (deflated 42%)
2022-11-23T02:59:34.3805163Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015011.xml (deflated 42%)
2022-11-23T02:59:34.3806239Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015019.xml (deflated 42%)
2022-11-23T02:59:34.3807080Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015021.xml (deflated 46%)
2022-11-23T02:59:34.3808026Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015028.xml (deflated 47%)
2022-11-23T02:59:34.3808855Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015035.xml (deflated 48%)
2022-11-23T02:59:34.3809708Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015042.xml (deflated 46%)
2022-11-23T02:59:34.3810553Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015049.xml (deflated 40%)
2022-11-23T02:59:34.3811395Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015056.xml (deflated 40%)
2022-11-23T02:59:34.3812223Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015102.xml (deflated 40%)
2022-11-23T02:59:34.3813079Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015109.xml (deflated 42%)
2022-11-23T02:59:34.3813919Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015116.xml (deflated 41%)
2022-11-23T02:59:34.3814773Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015123.xml (deflated 41%)
2022-11-23T02:59:34.3815602Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015129.xml (deflated 40%)
2022-11-23T02:59:34.3816456Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015136.xml (deflated 41%)
2022-11-23T02:59:34.3817290Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015139.xml (deflated 41%)
2022-11-23T02:59:34.3818134Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015141.xml (deflated 41%)
2022-11-23T02:59:34.3818962Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015148.xml (deflated 42%)
2022-11-23T02:59:34.3819819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015155.xml (deflated 42%)
2022-11-23T02:59:34.3820732Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015157.xml (deflated 42%)
2022-11-23T02:59:34.3821590Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015200.xml (deflated 42%)
2022-11-23T02:59:34.3822415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015202.xml (deflated 42%)
2022-11-23T02:59:34.3823260Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015204.xml (deflated 41%)
2022-11-23T02:59:34.3824120Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015211.xml (deflated 40%)
2022-11-23T02:59:34.3824960Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015218.xml (deflated 43%)
2022-11-23T02:59:34.3825796Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015221.xml (deflated 41%)
2022-11-23T02:59:34.3826639Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015228.xml (deflated 40%)
2022-11-23T02:59:34.3827481Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015234.xml (deflated 40%)
2022-11-23T02:59:34.3828329Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015241.xml (deflated 40%)
2022-11-23T02:59:34.3829259Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015248.xml (deflated 41%)
2022-11-23T02:59:34.3830108Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015255.xml (deflated 41%)
2022-11-23T02:59:34.3830957Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015302.xml (deflated 42%)
2022-11-23T02:59:34.3831802Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015309.xml (deflated 42%)
2022-11-23T02:59:34.3832648Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015315.xml (deflated 40%)
2022-11-23T02:59:34.3833470Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015322.xml (deflated 41%)
2022-11-23T02:59:34.3834321Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015329.xml (deflated 41%)
2022-11-23T02:59:34.3835172Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015336.xml (deflated 41%)
2022-11-23T02:59:34.3836015Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015342.xml (deflated 40%)
2022-11-23T02:59:34.3836840Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015349.xml (deflated 41%)
2022-11-23T02:59:34.3837689Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015356.xml (deflated 40%)
2022-11-23T02:59:34.3838523Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015403.xml (deflated 40%)
2022-11-23T02:59:34.3839374Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015410.xml (deflated 41%)
2022-11-23T02:59:34.3840260Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015416.xml (deflated 41%)
2022-11-23T02:59:34.3841101Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015423.xml (deflated 42%)
2022-11-23T02:59:34.3842003Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015430.xml (deflated 42%)
2022-11-23T02:59:34.3842861Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015437.xml (deflated 42%)
2022-11-23T02:59:34.3843693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015443.xml (deflated 42%)
2022-11-23T02:59:34.3844537Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015450.xml (deflated 41%)
2022-11-23T02:59:34.3845380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015457.xml (deflated 40%)
2022-11-23T02:59:34.3846455Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015504.xml (deflated 40%)
2022-11-23T02:59:34.3847293Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015512.xml (deflated 41%)
2022-11-23T02:59:34.3848146Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015519.xml (deflated 40%)
2022-11-23T02:59:34.3848989Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015526.xml (deflated 41%)
2022-11-23T02:59:34.3849925Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015534.xml (deflated 40%)
2022-11-23T02:59:34.3850749Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015541.xml (deflated 40%)
2022-11-23T02:59:34.3851599Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015548.xml (deflated 40%)
2022-11-23T02:59:34.3852446Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015554.xml (deflated 41%)
2022-11-23T02:59:34.3853382Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015603.xml (deflated 40%)
2022-11-23T02:59:34.3854227Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015612.xml (deflated 41%)
2022-11-23T02:59:34.3855063Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015621.xml (deflated 42%)
2022-11-23T02:59:34.3855891Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015623.xml (deflated 42%)
2022-11-23T02:59:34.3856736Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015625.xml (deflated 42%)
2022-11-23T02:59:34.3857593Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015628.xml (deflated 41%)
2022-11-23T02:59:34.3858429Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015630.xml (deflated 42%)
2022-11-23T02:59:34.3859252Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015633.xml (deflated 42%)
2022-11-23T02:59:34.3860105Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015635.xml (deflated 42%)
2022-11-23T02:59:34.3860948Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015637.xml (deflated 42%)
2022-11-23T02:59:34.3861788Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015640.xml (deflated 42%)
2022-11-23T02:59:34.3862607Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015642.xml (deflated 42%)
2022-11-23T02:59:34.3863518Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015645.xml (deflated 42%)
2022-11-23T02:59:34.3864374Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015647.xml (deflated 42%)
2022-11-23T02:59:34.3865223Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015649.xml (deflated 42%)
2022-11-23T02:59:34.3866048Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015652.xml (deflated 42%)
2022-11-23T02:59:34.3866904Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015654.xml (deflated 42%)
2022-11-23T02:59:34.3867740Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015657.xml (deflated 42%)
2022-11-23T02:59:34.3868587Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015659.xml (deflated 42%)
2022-11-23T02:59:34.3869429Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015701.xml (deflated 42%)
2022-11-23T02:59:34.3870266Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015704.xml (deflated 42%)
2022-11-23T02:59:34.3871179Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015706.xml (deflated 42%)
2022-11-23T02:59:34.3872013Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015709.xml (deflated 42%)
2022-11-23T02:59:34.3872852Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015711.xml (deflated 42%)
2022-11-23T02:59:34.3873678Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015713.xml (deflated 42%)
2022-11-23T02:59:34.3874521Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015716.xml (deflated 42%)
2022-11-23T02:59:34.3875357Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015718.xml (deflated 41%)
2022-11-23T02:59:34.3876211Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015726.xml (deflated 42%)
2022-11-23T02:59:34.3877037Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015733.xml (deflated 42%)
2022-11-23T02:59:34.3877884Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015735.xml (deflated 41%)
2022-11-23T02:59:34.3878733Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015743.xml (deflated 40%)
2022-11-23T02:59:34.3879575Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015752.xml (deflated 40%)
2022-11-23T02:59:34.3880398Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015759.xml (deflated 41%)
2022-11-23T02:59:34.3881255Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015806.xml (deflated 42%)
2022-11-23T02:59:34.3882097Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015813.xml (deflated 42%)
2022-11-23T02:59:34.3882949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015819.xml (deflated 40%)
2022-11-23T02:59:34.3883839Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015827.xml (deflated 40%)
2022-11-23T02:59:34.3884697Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015835.xml (deflated 42%)
2022-11-23T02:59:34.3885542Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015842.xml (deflated 40%)
2022-11-23T02:59:34.3886567Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015849.xml (deflated 40%)
2022-11-23T02:59:34.3887408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015856.xml (deflated 41%)
2022-11-23T02:59:34.3888255Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015858.xml (deflated 41%)
2022-11-23T02:59:34.3889097Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015900.xml (deflated 41%)
2022-11-23T02:59:34.3889949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015903.xml (deflated 41%)
2022-11-23T02:59:34.3890793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015905.xml (deflated 41%)
2022-11-23T02:59:34.3891636Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015908.xml (deflated 41%)
2022-11-23T02:59:34.3892573Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015910.xml (deflated 41%)
2022-11-23T02:59:34.3893411Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015912.xml (deflated 41%)
2022-11-23T02:59:34.3894245Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015915.xml (deflated 41%)
2022-11-23T02:59:34.3895098Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015922.xml (deflated 42%)
2022-11-23T02:59:34.3895941Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015926.xml (deflated 40%)
2022-11-23T02:59:34.3896779Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015932.xml (deflated 42%)
2022-11-23T02:59:34.3897611Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015939.xml (deflated 40%)
2022-11-23T02:59:34.3898453Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015947.xml (deflated 42%)
2022-11-23T02:59:34.3899301Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015951.xml (deflated 42%)
2022-11-23T02:59:34.3900144Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015955.xml (deflated 42%)
2022-11-23T02:59:34.3900970Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015959.xml (deflated 40%)
2022-11-23T02:59:34.3901819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020007.xml (deflated 40%)
2022-11-23T02:59:34.3902672Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020016.xml (deflated 42%)
2022-11-23T02:59:34.3903508Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020020.xml (deflated 42%)
2022-11-23T02:59:34.3904343Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020024.xml (deflated 40%)
2022-11-23T02:59:34.3905241Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020032.xml (deflated 40%)
2022-11-23T02:59:34.3906097Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020039.xml (deflated 40%)
2022-11-23T02:59:34.3906944Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020047.xml (deflated 40%)
2022-11-23T02:59:34.3907778Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020056.xml (deflated 42%)
2022-11-23T02:59:34.3908611Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020100.xml (deflated 41%)
2022-11-23T02:59:34.3909459Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020107.xml (deflated 42%)
2022-11-23T02:59:34.3910305Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020111.xml (deflated 41%)
2022-11-23T02:59:34.3911124Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020119.xml (deflated 42%)
2022-11-23T02:59:34.3911976Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020123.xml (deflated 42%)
2022-11-23T02:59:34.3912891Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020127.xml (deflated 40%)
2022-11-23T02:59:34.3913727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020136.xml (deflated 40%)
2022-11-23T02:59:34.3914573Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020144.xml (deflated 42%)
2022-11-23T02:59:34.3915415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020149.xml (deflated 40%)
2022-11-23T02:59:34.3916258Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020157.xml (deflated 41%)
2022-11-23T02:59:34.3917098Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020205.xml (deflated 41%)
2022-11-23T02:59:34.3917938Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020214.xml (deflated 41%)
2022-11-23T02:59:34.3918774Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020222.xml (deflated 41%)
2022-11-23T02:59:34.3919620Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020230.xml (deflated 41%)
2022-11-23T02:59:34.3920466Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020239.xml (deflated 41%)
2022-11-23T02:59:34.3921313Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020247.xml (deflated 41%)
2022-11-23T02:59:34.3922136Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020255.xml (deflated 41%)
2022-11-23T02:59:34.3922982Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020304.xml (deflated 40%)
2022-11-23T02:59:34.3923826Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020312.xml (deflated 41%)
2022-11-23T02:59:34.3924666Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020320.xml (deflated 41%)
2022-11-23T02:59:34.3925490Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020329.xml (deflated 41%)
2022-11-23T02:59:34.3926645Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020337.xml (deflated 42%)
2022-11-23T02:59:34.3927494Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020341.xml (deflated 41%)
2022-11-23T02:59:34.3928347Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020350.xml (deflated 40%)
2022-11-23T02:59:34.3929216Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020358.xml (deflated 40%)
2022-11-23T02:59:34.3930063Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020405.xml (deflated 40%)
2022-11-23T02:59:34.3930906Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020413.xml (deflated 40%)
2022-11-23T02:59:34.3931763Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020421.xml (deflated 40%)
2022-11-23T02:59:34.3932594Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020438.xml (deflated 41%)
2022-11-23T02:59:34.3933436Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020447.xml (deflated 41%)
2022-11-23T02:59:34.3934367Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020455.xml (deflated 41%)
2022-11-23T02:59:34.3935205Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020503.xml (deflated 42%)
2022-11-23T02:59:34.3936040Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020511.xml (deflated 42%)
2022-11-23T02:59:34.3936897Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020515.xml (deflated 42%)
2022-11-23T02:59:34.3937735Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020519.xml (deflated 41%)
2022-11-23T02:59:34.3938583Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020528.xml (deflated 41%)
2022-11-23T02:59:34.3939413Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020535.xml (deflated 42%)
2022-11-23T02:59:34.3940273Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020539.xml (deflated 40%)
2022-11-23T02:59:34.3941109Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020548.xml (deflated 42%)
2022-11-23T02:59:34.3941954Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020552.xml (deflated 41%)
2022-11-23T02:59:34.3942796Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020600.xml (deflated 41%)
2022-11-23T02:59:34.3943637Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020608.xml (deflated 40%)
2022-11-23T02:59:34.3944489Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020616.xml (deflated 42%)
2022-11-23T02:59:34.3945330Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020620.xml (deflated 42%)
2022-11-23T02:59:34.3946160Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020624.xml (deflated 42%)
2022-11-23T02:59:34.3947055Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020628.xml (deflated 40%)
2022-11-23T02:59:34.3947905Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020636.xml (deflated 41%)
2022-11-23T02:59:34.3948743Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020644.xml (deflated 40%)
2022-11-23T02:59:34.3949566Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020651.xml (deflated 40%)
2022-11-23T02:59:34.3950416Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020658.xml (deflated 42%)
2022-11-23T02:59:34.3951263Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020702.xml (deflated 42%)
2022-11-23T02:59:34.3952107Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020706.xml (deflated 40%)
2022-11-23T02:59:34.3953226Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020713.xml (deflated 41%)
2022-11-23T02:59:34.3954050Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020720.xml (deflated 40%)
2022-11-23T02:59:34.3954893Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020727.xml (deflated 42%)
2022-11-23T02:59:34.3955831Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020729.xml (deflated 40%)
2022-11-23T02:59:34.3956670Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020736.xml (deflated 41%)
2022-11-23T02:59:34.3957506Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020743.xml (deflated 40%)
2022-11-23T02:59:34.3958355Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020749.xml (deflated 42%)
2022-11-23T02:59:34.3959199Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020753.xml (deflated 41%)
2022-11-23T02:59:34.3960037Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020800.xml (deflated 41%)
2022-11-23T02:59:34.3960873Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020807.xml (deflated 41%)
2022-11-23T02:59:34.3961716Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020814.xml (deflated 40%)
2022-11-23T02:59:34.3962560Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020821.xml (deflated 40%)
2022-11-23T02:59:34.3963402Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020828.xml (deflated 40%)
2022-11-23T02:59:34.3964232Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020836.xml (deflated 41%)
2022-11-23T02:59:34.3965077Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020843.xml (deflated 41%)
2022-11-23T02:59:34.3966090Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020849.xml (deflated 40%)
2022-11-23T02:59:34.3966938Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020856.xml (deflated 41%)
2022-11-23T02:59:34.3967764Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020903.xml (deflated 41%)
2022-11-23T02:59:34.3968677Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020928.xml (deflated 40%)
2022-11-23T02:59:34.3969533Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020952.xml (deflated 42%)
2022-11-23T02:59:34.3970372Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020958.xml (deflated 40%)
2022-11-23T02:59:34.3971208Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021007.xml (deflated 40%)
2022-11-23T02:59:34.3972049Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021014.xml (deflated 40%)
2022-11-23T02:59:34.3972888Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021021.xml (deflated 41%)
2022-11-23T02:59:34.3973733Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021028.xml (deflated 42%)
2022-11-23T02:59:34.3974562Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021030.xml (deflated 42%)
2022-11-23T02:59:34.3975409Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021033.xml (deflated 42%)
2022-11-23T02:59:34.3976338Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021035.xml (deflated 42%)
2022-11-23T02:59:34.3977179Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021038.xml (deflated 42%)
2022-11-23T02:59:34.3978002Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021040.xml (deflated 42%)
2022-11-23T02:59:34.3978851Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021042.xml (deflated 42%)
2022-11-23T02:59:34.3979691Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021045.xml (deflated 42%)
2022-11-23T02:59:34.3980530Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021047.xml (deflated 40%)
2022-11-23T02:59:34.3981354Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021054.xml (deflated 40%)
2022-11-23T02:59:34.3982207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021101.xml (deflated 42%)
2022-11-23T02:59:34.3983051Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021103.xml (deflated 42%)
2022-11-23T02:59:34.3983890Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021105.xml (deflated 42%)
2022-11-23T02:59:34.3984717Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021109.xml (deflated 40%)
2022-11-23T02:59:34.3985565Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021118.xml (deflated 41%)
2022-11-23T02:59:34.3986406Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021126.xml (deflated 41%)
2022-11-23T02:59:34.3987253Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021134.xml (deflated 42%)
2022-11-23T02:59:34.3988097Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021138.xml (deflated 41%)
2022-11-23T02:59:34.3988923Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021142.xml (deflated 40%)
2022-11-23T02:59:34.3989816Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021149.xml (deflated 40%)
2022-11-23T02:59:34.3990666Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021155.xml (deflated 42%)
2022-11-23T02:59:34.3991506Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021200.xml (deflated 41%)
2022-11-23T02:59:34.3992339Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021207.xml (deflated 41%)
2022-11-23T02:59:34.3993181Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021213.xml (deflated 41%)
2022-11-23T02:59:34.3994021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021220.xml (deflated 41%)
2022-11-23T02:59:34.3994868Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021227.xml (deflated 42%)
2022-11-23T02:59:34.3995699Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021234.xml (deflated 42%)
2022-11-23T02:59:34.3996543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021240.xml (deflated 42%)
2022-11-23T02:59:34.3997447Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021247.xml (deflated 42%)
2022-11-23T02:59:34.3998291Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021254.xml (deflated 41%)
2022-11-23T02:59:34.3999116Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021301.xml (deflated 41%)
2022-11-23T02:59:34.3999964Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021308.xml (deflated 42%)
2022-11-23T02:59:34.4000808Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021310.xml (deflated 42%)
2022-11-23T02:59:34.4001646Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021317.xml (deflated 43%)
2022-11-23T02:59:34.4002480Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021319.xml (deflated 43%)
2022-11-23T02:59:34.4003319Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021322.xml (deflated 41%)
2022-11-23T02:59:34.4004161Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021329.xml (deflated 42%)
2022-11-23T02:59:34.4005006Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021331.xml (deflated 42%)
2022-11-23T02:59:34.4006000Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021333.xml (deflated 41%)
2022-11-23T02:59:34.4006855Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021340.xml (deflated 41%)
2022-11-23T02:59:34.4007701Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021347.xml (deflated 41%)
2022-11-23T02:59:34.4008543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021354.xml (deflated 40%)
2022-11-23T02:59:34.4009378Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021401.xml (deflated 42%)
2022-11-23T02:59:34.4010290Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021403.xml (deflated 41%)
2022-11-23T02:59:34.4011148Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021405.xml (deflated 40%)
2022-11-23T02:59:34.4011991Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021412.xml (deflated 42%)
2022-11-23T02:59:34.4012819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021419.xml (deflated 41%)
2022-11-23T02:59:34.4013669Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021426.xml (deflated 40%)
2022-11-23T02:59:34.4014510Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021432.xml (deflated 40%)
2022-11-23T02:59:34.4015348Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021439.xml (deflated 40%)
2022-11-23T02:59:34.4016180Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021446.xml (deflated 40%)
2022-11-23T02:59:34.4017020Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021453.xml (deflated 41%)
2022-11-23T02:59:34.4017862Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021500.xml (deflated 41%)
2022-11-23T02:59:34.4018783Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021502.xml (deflated 41%)
2022-11-23T02:59:34.4019612Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021505.xml (deflated 41%)
2022-11-23T02:59:34.4020452Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021507.xml (deflated 41%)
2022-11-23T02:59:34.4021298Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021514.xml (deflated 41%)
2022-11-23T02:59:34.4022145Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021521.xml (deflated 41%)
2022-11-23T02:59:34.4022973Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021528.xml (deflated 41%)
2022-11-23T02:59:34.4023822Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021535.xml (deflated 41%)
2022-11-23T02:59:34.4024661Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021542.xml (deflated 40%)
2022-11-23T02:59:34.4025499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021549.xml (deflated 41%)
2022-11-23T02:59:34.4026327Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021557.xml (deflated 41%)
2022-11-23T02:59:34.4027174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021604.xml (deflated 41%)
2022-11-23T02:59:34.4028011Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021612.xml (deflated 41%)
2022-11-23T02:59:34.4028856Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021620.xml (deflated 40%)
2022-11-23T02:59:34.4029723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021633.xml (deflated 40%)
2022-11-23T02:59:34.4030561Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_traverse/TEST-TestTraverse-20221123021720.xml (deflated 77%)
2022-11-23T02:59:34.4031397Z   adding: test/test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20221123021728.xml (deflated 83%)
2022-11-23T02:59:34.4032193Z   adding: test/test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20221123021731.xml (deflated 62%)
2022-11-23T02:59:34.4033001Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_utils/TEST-TestMedatadaIndex-20221123021735.xml (deflated 71%)
2022-11-23T02:59:34.4033846Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20221123021739.xml (deflated 54%)
2022-11-23T02:59:34.4034724Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20221123021743.xml (deflated 68%)
2022-11-23T02:59:34.4035575Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20221123021749.xml (deflated 83%)
2022-11-23T02:59:34.4036439Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20221123021749.xml (deflated 90%)
2022-11-23T02:59:34.4037314Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20221123021756.xml (deflated 41%)
2022-11-23T02:59:34.4038154Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_uneven/TEST-TestUnevenParamShard-20221123021803.xml (deflated 41%)
2022-11-23T02:59:34.4039043Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221123021811.xml (deflated 47%)
2022-11-23T02:59:34.4040027Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20221123021819.xml (deflated 60%)
2022-11-23T02:59:34.4040947Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20221123021827.xml (deflated 60%)
2022-11-23T02:59:34.4041835Z   adding: test/test-reports/python-unittest/distributed.test_c10d_error_logger/TEST-C10dErrorLoggerTest-20221123021836.xml (deflated 52%)
2022-11-23T02:59:34.4042721Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20221123021845.xml (deflated 66%)
2022-11-23T02:59:34.4043624Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20221123021845.xml (deflated 79%)
2022-11-23T02:59:34.4044537Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20221123021857.xml (deflated 68%)
2022-11-23T02:59:34.4045474Z   adding: test/test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api/TEST-TensorParallelAPITests-20221123021908.xml (deflated 73%)
2022-11-23T02:59:34.4046515Z   adding: test/test-reports/python-unittest/distributed._tensor.test_math_ops/TEST-DistMathOpsTest-20221123021922.xml (deflated 61%)
2022-11-23T02:59:34.4047361Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20221123021936.xml (deflated 76%)
2022-11-23T02:59:34.4048218Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20221123021952.xml (deflated 75%)
2022-11-23T02:59:34.4049069Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_memory/TEST-TestFSDPMemory-20221123022007.xml (deflated 55%)
2022-11-23T02:59:34.4049944Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20221123022024.xml (deflated 80%)
2022-11-23T02:59:34.4050839Z   adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPInitialization-20221123022044.xml (deflated 68%)
2022-11-23T02:59:34.4051684Z   adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPRuntime-20221123022044.xml (deflated 39%)
2022-11-23T02:59:34.4052638Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123022109.xml (deflated 43%)
2022-11-23T02:59:34.4053564Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123022112.xml (deflated 42%)
2022-11-23T02:59:34.4054474Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123022120.xml (deflated 43%)
2022-11-23T02:59:34.4055366Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123022128.xml (deflated 42%)
2022-11-23T02:59:34.4056270Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123022136.xml (deflated 42%)
2022-11-23T02:59:34.4057163Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20221123022144.xml (deflated 42%)
2022-11-23T02:59:34.4058047Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20221123022151.xml (deflated 55%)
2022-11-23T02:59:34.4058941Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20221123022219.xml (deflated 55%)
2022-11-23T02:59:34.4059863Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20221123022219.xml (deflated 81%)
2022-11-23T02:59:34.4060941Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20221123022219.xml (deflated 80%)
2022-11-23T02:59:34.4061879Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20221123022219.xml (deflated 64%)
2022-11-23T02:59:34.4062777Z   adding: test/test-reports/python-unittest/distributed._tensor.test_matrix_ops/TEST-DistMatrixOpsTest-20221123022249.xml (deflated 75%)
2022-11-23T02:59:34.4063664Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20221123022321.xml (deflated 83%)
2022-11-23T02:59:34.4064659Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123022359.xml (deflated 43%)
2022-11-23T02:59:34.4065725Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123022403.xml (deflated 44%)
2022-11-23T02:59:34.4066794Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20221123022407.xml (deflated 43%)
2022-11-23T02:59:34.4067796Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022411.xml (deflated 41%)
2022-11-23T02:59:34.4068779Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022418.xml (deflated 42%)
2022-11-23T02:59:34.4069757Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022426.xml (deflated 42%)
2022-11-23T02:59:34.4070726Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022433.xml (deflated 42%)
2022-11-23T02:59:34.4071687Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022440.xml (deflated 42%)
2022-11-23T02:59:34.4072656Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022448.xml (deflated 41%)
2022-11-23T02:59:34.4073718Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022455.xml (deflated 41%)
2022-11-23T02:59:34.4074728Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20221123022502.xml (deflated 42%)
2022-11-23T02:59:34.4075715Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022513.xml (deflated 42%)
2022-11-23T02:59:34.4076682Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022521.xml (deflated 42%)
2022-11-23T02:59:34.4077657Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022528.xml (deflated 42%)
2022-11-23T02:59:34.4078613Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022536.xml (deflated 43%)
2022-11-23T02:59:34.4079580Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022543.xml (deflated 42%)
2022-11-23T02:59:34.4080555Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022550.xml (deflated 42%)
2022-11-23T02:59:34.4081526Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022558.xml (deflated 42%)
2022-11-23T02:59:34.4082479Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022605.xml (deflated 42%)
2022-11-23T02:59:34.4083521Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20221123022613.xml (deflated 42%)
2022-11-23T02:59:34.4084429Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20221123022619.xml (deflated 93%)
2022-11-23T02:59:34.4085256Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022711.xml (deflated 40%)
2022-11-23T02:59:34.4086248Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022715.xml (deflated 39%)
2022-11-23T02:59:34.4087067Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022719.xml (deflated 39%)
2022-11-23T02:59:34.4087874Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20221123022723.xml (deflated 39%)
2022-11-23T02:59:34.4088676Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022727.xml (deflated 39%)
2022-11-23T02:59:34.4089469Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20221123022731.xml (deflated 39%)
2022-11-23T02:59:34.4090299Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022735.xml (deflated 40%)
2022-11-23T02:59:34.4091153Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20221123022739.xml (deflated 40%)
2022-11-23T02:59:34.4091992Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20221123022742.xml (deflated 40%)
2022-11-23T02:59:34.4092816Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022744.xml (deflated 39%)
2022-11-23T02:59:34.4093667Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20221123022748.xml (deflated 39%)
2022-11-23T02:59:34.4094504Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20221123022752.xml (deflated 39%)
2022-11-23T02:59:34.4095336Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20221123022756.xml (deflated 39%)
2022-11-23T02:59:34.4096157Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022800.xml (deflated 40%)
2022-11-23T02:59:34.4096998Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20221123022804.xml (deflated 39%)
2022-11-23T02:59:34.4097909Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022808.xml (deflated 39%)
2022-11-23T02:59:34.4098765Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022812.xml (deflated 39%)
2022-11-23T02:59:34.4099579Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022816.xml (deflated 39%)
2022-11-23T02:59:34.4100424Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20221123022819.xml (deflated 40%)
2022-11-23T02:59:34.4101246Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123022834.xml (deflated 38%)
2022-11-23T02:59:34.4102067Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20221123022837.xml (deflated 39%)
2022-11-23T02:59:34.4102863Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022841.xml (deflated 38%)
2022-11-23T02:59:34.4103668Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022845.xml (deflated 39%)
2022-11-23T02:59:34.4104467Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022849.xml (deflated 38%)
2022-11-23T02:59:34.4105266Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022853.xml (deflated 38%)
2022-11-23T02:59:34.4106155Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022857.xml (deflated 38%)
2022-11-23T02:59:34.4106959Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022901.xml (deflated 39%)
2022-11-23T02:59:34.4107758Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022905.xml (deflated 38%)
2022-11-23T02:59:34.4108545Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20221123022911.xml (deflated 39%)
2022-11-23T02:59:34.4109369Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20221123022915.xml (deflated 81%)
2022-11-23T02:59:34.4110194Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20221123022915.xml (deflated 89%)
2022-11-23T02:59:34.4111132Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20221123023042.xml (deflated 43%)
2022-11-23T02:59:34.4112158Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20221123023042.xml (deflated 43%)
2022-11-23T02:59:34.4113114Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20221123023042.xml (deflated 60%)
2022-11-23T02:59:34.4114062Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20221123023042.xml (deflated 58%)
2022-11-23T02:59:34.4115012Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20221123023042.xml (deflated 58%)
2022-11-23T02:59:34.4115964Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20221123023042.xml (deflated 60%)
2022-11-23T02:59:34.4116898Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20221123023042.xml (deflated 61%)
2022-11-23T02:59:34.4117866Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20221123023042.xml (deflated 89%)
2022-11-23T02:59:34.4118877Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20221123023042.xml (deflated 69%)
2022-11-23T02:59:34.4119947Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20221123023042.xml (deflated 87%)
2022-11-23T02:59:34.4120990Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20221123023042.xml (deflated 82%)
2022-11-23T02:59:34.4122054Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20221123023042.xml (deflated 61%)
2022-11-23T02:59:34.4123082Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20221123023314.xml (deflated 54%)
2022-11-23T02:59:34.4124139Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20221123023314.xml (deflated 83%)
2022-11-23T02:59:34.4125222Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20221123023314.xml (deflated 45%)
2022-11-23T02:59:34.4126450Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20221123023314.xml (deflated 76%)
2022-11-23T02:59:34.4127502Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20221123023314.xml (deflated 63%)
2022-11-23T02:59:34.4128623Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221123023604.xml (deflated 92%)
2022-11-23T02:59:34.4129689Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221123023604.xml (deflated 64%)
2022-11-23T02:59:34.4130725Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20221123024021.xml (deflated 41%)
2022-11-23T02:59:34.4131777Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123024028.xml (deflated 41%)
2022-11-23T02:59:34.4132816Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123024037.xml (deflated 41%)
2022-11-23T02:59:34.4133853Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123024046.xml (deflated 41%)
2022-11-23T02:59:34.4134865Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024054.xml (deflated 41%)
2022-11-23T02:59:34.4135887Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024102.xml (deflated 41%)
2022-11-23T02:59:34.4136914Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024110.xml (deflated 41%)
2022-11-23T02:59:34.4137938Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024118.xml (deflated 41%)
2022-11-23T02:59:34.4138936Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20221123024126.xml (deflated 40%)
2022-11-23T02:59:34.4139949Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024136.xml (deflated 40%)
2022-11-23T02:59:34.4140956Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024143.xml (deflated 40%)
2022-11-23T02:59:34.4141957Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024149.xml (deflated 40%)
2022-11-23T02:59:34.4143023Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024156.xml (deflated 40%)
2022-11-23T02:59:34.4144018Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024204.xml (deflated 40%)
2022-11-23T02:59:34.4145011Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024211.xml (deflated 41%)
2022-11-23T02:59:34.4146008Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024218.xml (deflated 41%)
2022-11-23T02:59:34.4147001Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024225.xml (deflated 40%)
2022-11-23T02:59:34.4148037Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024231.xml (deflated 42%)
2022-11-23T02:59:34.4149132Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024242.xml (deflated 43%)
2022-11-23T02:59:34.4150227Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024255.xml (deflated 42%)
2022-11-23T02:59:34.4151392Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024308.xml (deflated 43%)
2022-11-23T02:59:34.4152489Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024319.xml (deflated 43%)
2022-11-23T02:59:34.4153569Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024330.xml (deflated 43%)
2022-11-23T02:59:34.4154666Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024342.xml (deflated 43%)
2022-11-23T02:59:34.4155753Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024354.xml (deflated 44%)
2022-11-23T02:59:34.4156848Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024405.xml (deflated 43%)
2022-11-23T02:59:34.4157921Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024417.xml (deflated 43%)
2022-11-23T02:59:34.4159014Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024424.xml (deflated 43%)
2022-11-23T02:59:34.4160101Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024431.xml (deflated 43%)
2022-11-23T02:59:34.4161194Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024438.xml (deflated 43%)
2022-11-23T02:59:34.4162289Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024446.xml (deflated 43%)
2022-11-23T02:59:34.4163370Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024454.xml (deflated 43%)
2022-11-23T02:59:34.4164457Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024502.xml (deflated 42%)
2022-11-23T02:59:34.4165594Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024514.xml (deflated 43%)
2022-11-23T02:59:34.4166877Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024518.xml (deflated 43%)
2022-11-23T02:59:34.4167952Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024539.xml (deflated 43%)
2022-11-23T02:59:34.4169048Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024553.xml (deflated 42%)
2022-11-23T02:59:34.4170136Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024605.xml (deflated 42%)
2022-11-23T02:59:34.4171231Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024613.xml (deflated 42%)
2022-11-23T02:59:34.4172306Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024622.xml (deflated 42%)
2022-11-23T02:59:34.4173398Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024631.xml (deflated 42%)
2022-11-23T02:59:34.4174585Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024640.xml (deflated 43%)
2022-11-23T02:59:34.4175669Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024645.xml (deflated 42%)
2022-11-23T02:59:34.4176753Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024655.xml (deflated 42%)
2022-11-23T02:59:34.4177831Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024705.xml (deflated 42%)
2022-11-23T02:59:34.4178926Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024716.xml (deflated 42%)
2022-11-23T02:59:34.4180019Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024726.xml (deflated 42%)
2022-11-23T02:59:34.4181120Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024736.xml (deflated 42%)
2022-11-23T02:59:34.4182195Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024747.xml (deflated 42%)
2022-11-23T02:59:34.4183294Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024757.xml (deflated 42%)
2022-11-23T02:59:34.4184385Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024808.xml (deflated 42%)
2022-11-23T02:59:34.4185475Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024818.xml (deflated 42%)
2022-11-23T02:59:34.4186565Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024829.xml (deflated 42%)
2022-11-23T02:59:34.4187639Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024839.xml (deflated 42%)
2022-11-23T02:59:34.4188788Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024850.xml (deflated 42%)
2022-11-23T02:59:34.4189895Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024900.xml (deflated 42%)
2022-11-23T02:59:34.4190987Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024910.xml (deflated 42%)
2022-11-23T02:59:34.4192070Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024921.xml (deflated 42%)
2022-11-23T02:59:34.4193163Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024931.xml (deflated 42%)
2022-11-23T02:59:34.4194257Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024941.xml (deflated 43%)
2022-11-23T02:59:34.4195343Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024951.xml (deflated 42%)
2022-11-23T02:59:34.4196434Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025000.xml (deflated 42%)
2022-11-23T02:59:34.4197504Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025010.xml (deflated 43%)
2022-11-23T02:59:34.4198666Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025020.xml (deflated 43%)
2022-11-23T02:59:34.4199763Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025031.xml (deflated 43%)
2022-11-23T02:59:34.4200854Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025038.xml (deflated 43%)
2022-11-23T02:59:34.4201935Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025045.xml (deflated 43%)
2022-11-23T02:59:34.4203023Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025052.xml (deflated 42%)
2022-11-23T02:59:34.4204113Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025100.xml (deflated 42%)
2022-11-23T02:59:34.4205197Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025108.xml (deflated 42%)
2022-11-23T02:59:34.4206464Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025117.xml (deflated 42%)
2022-11-23T02:59:34.4207545Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025125.xml (deflated 42%)
2022-11-23T02:59:34.4208626Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025134.xml (deflated 42%)
2022-11-23T02:59:34.4209715Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025142.xml (deflated 42%)
2022-11-23T02:59:34.4210807Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025151.xml (deflated 42%)
2022-11-23T02:59:34.4211873Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025159.xml (deflated 42%)
2022-11-23T02:59:34.4213034Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025210.xml (deflated 42%)
2022-11-23T02:59:34.4214134Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025220.xml (deflated 43%)
2022-11-23T02:59:34.4215221Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025227.xml (deflated 43%)
2022-11-23T02:59:34.4216295Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025238.xml (deflated 43%)
2022-11-23T02:59:34.4217391Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025245.xml (deflated 43%)
2022-11-23T02:59:34.4218486Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025252.xml (deflated 43%)
2022-11-23T02:59:34.4219573Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025259.xml (deflated 42%)
2022-11-23T02:59:34.4220659Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025308.xml (deflated 42%)
2022-11-23T02:59:34.4221814Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025316.xml (deflated 42%)
2022-11-23T02:59:34.4222903Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025323.xml (deflated 42%)
2022-11-23T02:59:34.4223991Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025335.xml (deflated 42%)
2022-11-23T02:59:34.4225083Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025348.xml (deflated 42%)
2022-11-23T02:59:34.4226153Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025402.xml (deflated 42%)
2022-11-23T02:59:34.4227246Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025415.xml (deflated 42%)
2022-11-23T02:59:34.4228336Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025433.xml (deflated 42%)
2022-11-23T02:59:34.4229467Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025453.xml (deflated 42%)
2022-11-23T02:59:34.4230563Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025457.xml (deflated 42%)
2022-11-23T02:59:34.4231638Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025518.xml (deflated 42%)
2022-11-23T02:59:34.4232727Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025536.xml (deflated 42%)
2022-11-23T02:59:34.4233818Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025554.xml (deflated 42%)
2022-11-23T02:59:34.4234903Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025612.xml (deflated 42%)
2022-11-23T02:59:34.4236026Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025631.xml (deflated 42%)
2022-11-23T02:59:34.4237128Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025649.xml (deflated 42%)
2022-11-23T02:59:34.4238217Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025708.xml (deflated 42%)
2022-11-23T02:59:34.4239307Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025728.xml (deflated 42%)
2022-11-23T02:59:34.4240397Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025747.xml (deflated 42%)
2022-11-23T02:59:34.4241476Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025806.xml (deflated 43%)
2022-11-23T02:59:34.4242564Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025816.xml (deflated 43%)
2022-11-23T02:59:34.4243667Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123025827.xml (deflated 44%)
2022-11-23T02:59:34.4244856Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123025834.xml (deflated 44%)
2022-11-23T02:59:34.4246127Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123025841.xml (deflated 43%)
2022-11-23T02:59:34.4247025Z   adding: test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml (deflated 71%)
2022-11-23T02:59:34.4247729Z   adding: test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml (deflated 72%)
2022-11-23T02:59:34.4248428Z   adding: test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml (deflated 81%)
2022-11-23T02:59:34.4249142Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml (deflated 80%)
2022-11-23T02:59:34.4249889Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml (deflated 79%)
2022-11-23T02:59:34.4250664Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml (deflated 68%)
2022-11-23T02:59:34.4251356Z   adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%)
2022-11-23T02:59:34.4273026Z ##[group]Run # Remove any previous test reports if they exist
2022-11-23T02:59:34.4273489Z # Remove any previous test reports if they exist
2022-11-23T02:59:34.4273868Z rm -f usage-log-*.zip
2022-11-23T02:59:34.4274306Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-11-23T02:59:34.4274771Z # so check to see if the file exists first
2022-11-23T02:59:34.4275133Z if [ -f 'usage_log.txt' ]; then
2022-11-23T02:59:34.4275532Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-11-23T02:59:34.4275886Z fi
2022-11-23T02:59:34.4287812Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:59:34.4288177Z env:
2022-11-23T02:59:34.4288493Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:34.4288807Z   GPU_FLAG: --gpus all
2022-11-23T02:59:34.4289253Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:34.4289818Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655199096
2022-11-23T02:59:34.4290214Z ##[endgroup]
2022-11-23T02:59:34.5179108Z   adding: usage_log.txt (deflated 95%)
2022-11-23T02:59:34.5252822Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T02:59:34.5253180Z with:
2022-11-23T02:59:34.5253514Z   s3-prefix: pytorch/pytorch/3528293554/1/artifact

2022-11-23T02:59:34.5253970Z   retention-days: 14
2022-11-23T02:59:34.5254315Z   if-no-files-found: warn
2022-11-23T02:59:34.5254644Z   path: test-jsons-*.zip
2022-11-23T02:59:34.5254970Z   name: artifact
2022-11-23T02:59:34.5255289Z   s3-bucket: gha-artifacts
2022-11-23T02:59:34.5255622Z   region: us-east-1
2022-11-23T02:59:34.5255907Z env:
2022-11-23T02:59:34.5256211Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:34.5256538Z   GPU_FLAG: --gpus all
2022-11-23T02:59:34.5256973Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:34.5257402Z ##[endgroup]
2022-11-23T02:59:34.9673594Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T02:59:34.9674811Z With the provided path, there will be 1 file uploaded
2022-11-23T02:59:34.9675610Z Uploading to s3 prefix: pytorch/pytorch/3528293554/1/artifact
2022-11-23T02:59:34.9689969Z Starting upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655199096.zip
2022-11-23T02:59:35.0903373Z Finished upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655199096.zip
2022-11-23T02:59:35.1083528Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T02:59:35.1084125Z with:
2022-11-23T02:59:35.1084681Z   s3-prefix: pytorch/pytorch/3528293554/1/artifact

2022-11-23T02:59:35.1085298Z   retention-days: 14
2022-11-23T02:59:35.1086246Z   if-no-files-found: error
2022-11-23T02:59:35.1087001Z   path: test-reports-*.zip
2022-11-23T02:59:35.1087540Z   name: artifact
2022-11-23T02:59:35.1088063Z   s3-bucket: gha-artifacts
2022-11-23T02:59:35.1088589Z   region: us-east-1
2022-11-23T02:59:35.1089078Z env:
2022-11-23T02:59:35.1089582Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:35.1090112Z   GPU_FLAG: --gpus all
2022-11-23T02:59:35.1090891Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:35.1091642Z ##[endgroup]
2022-11-23T02:59:35.5512192Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T02:59:35.5512880Z With the provided path, there will be 1 file uploaded
2022-11-23T02:59:35.5513330Z Uploading to s3 prefix: pytorch/pytorch/3528293554/1/artifact
2022-11-23T02:59:35.5524335Z Starting upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655199096.zip
2022-11-23T02:59:35.7676998Z Finished upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655199096.zip
2022-11-23T02:59:35.7829585Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T02:59:35.7829946Z with:
2022-11-23T02:59:35.7830268Z   s3-prefix: pytorch/pytorch/3528293554/1/artifact

2022-11-23T02:59:35.7830628Z   retention-days: 14
2022-11-23T02:59:35.7830960Z   if-no-files-found: ignore
2022-11-23T02:59:35.7831274Z   path: usage-log-*.zip
2022-11-23T02:59:35.7831586Z   name: artifact
2022-11-23T02:59:35.7831890Z   s3-bucket: gha-artifacts
2022-11-23T02:59:35.7832197Z   region: us-east-1
2022-11-23T02:59:35.7832491Z env:
2022-11-23T02:59:35.7832789Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:35.7833098Z   GPU_FLAG: --gpus all
2022-11-23T02:59:35.7833553Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:35.7833991Z ##[endgroup]
2022-11-23T02:59:36.2357009Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T02:59:36.2357932Z With the provided path, there will be 1 file uploaded
2022-11-23T02:59:36.2358598Z Uploading to s3 prefix: pytorch/pytorch/3528293554/1/artifact
2022-11-23T02:59:36.2369985Z Starting upload of usage-log-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655199096.zip
2022-11-23T02:59:36.4157462Z Finished upload of usage-log-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655199096.zip
2022-11-23T02:59:36.4468250Z ##[group]Run # shellcheck disable=SC2156
2022-11-23T02:59:36.4468732Z # shellcheck disable=SC2156
2022-11-23T02:59:36.4469209Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2022-11-23T02:59:36.4482634Z shell: /usr/bin/bash -e {0}
2022-11-23T02:59:36.4482957Z env:
2022-11-23T02:59:36.4483243Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:36.4483578Z   GPU_FLAG: --gpus all
2022-11-23T02:59:36.4484037Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:36.4484450Z ##[endgroup]
2022-11-23T02:59:36.7496659Z ##[group]Run set -x
2022-11-23T02:59:36.7497025Z set -x
2022-11-23T02:59:36.7497385Z python3 -m pip install -r requirements.txt
2022-11-23T02:59:36.7497809Z python3 -m pip install boto3==1.19.12
2022-11-23T02:59:36.7498279Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-11-23T02:59:36.7510946Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:59:36.7511304Z env:
2022-11-23T02:59:36.7511624Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:36.7511950Z   GPU_FLAG: --gpus all
2022-11-23T02:59:36.7512416Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:36.7512889Z   AWS_DEFAULT_REGION: us-east-1
2022-11-23T02:59:36.7513208Z   BRANCH: master
2022-11-23T02:59:36.7513540Z   TEST_CONFIG: distributed
2022-11-23T02:59:36.7513870Z   SHARD_NUMBER: 1
2022-11-23T02:59:36.7514264Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T02:59:36.7514668Z   PR_NUMBER: 
2022-11-23T02:59:36.7515001Z   PYTORCH_RETRY_TEST_CASES: 1
2022-11-23T02:59:36.7515345Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-11-23T02:59:36.7516005Z   SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T02:59:36.7516458Z   TAG: 
2022-11-23T02:59:36.7516756Z   WORKFLOW_ID: 3528293554
2022-11-23T02:59:36.7517363Z   GITHUB_TOKEN: ***
2022-11-23T02:59:36.7517777Z   GHA_WORKFLOW_JOB_ID: 9655199096
2022-11-23T02:59:36.7518119Z ##[endgroup]
2022-11-23T02:59:36.7548668Z + python3 -m pip install -r requirements.txt
2022-11-23T02:59:37.0469242Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T02:59:37.0828886Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3)
2022-11-23T02:59:37.0868213Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.4)
2022-11-23T02:59:37.0879991Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2)
2022-11-23T02:59:37.0892744Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.58.0)
2022-11-23T02:59:37.1426820Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6)
2022-11-23T02:59:37.1439511Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2022-11-23T02:59:37.1549017Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0)
2022-11-23T02:59:37.1560563Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2022-11-23T02:59:37.1814010Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2022-11-23T02:59:37.2057257Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0)
2022-11-23T02:59:37.2069279Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6)
2022-11-23T02:59:37.2077753Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.4.0)
2022-11-23T02:59:37.2092623Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1)
2022-11-23T02:59:37.2118720Z Requirement already satisfied: filelock in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 15)) (3.8.0)
2022-11-23T02:59:37.2222154Z Requirement already satisfied: networkx in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 16)) (2.6.3)
2022-11-23T02:59:37.2454366Z Requirement already satisfied: jinja2 in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 17)) (3.1.2)
2022-11-23T02:59:37.2487311Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.38.4)
2022-11-23T02:59:37.2512322Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.1.0)
2022-11-23T02:59:37.2874828Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0)
2022-11-23T02:59:37.2889882Z Requirement already satisfied: exceptiongroup>=1.0.0; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.0.4)
2022-11-23T02:59:37.2913558Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2022-11-23T02:59:37.2930224Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.12)
2022-11-23T02:59:37.3151637Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24)
2022-11-23T02:59:37.3164635Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2022-11-23T02:59:37.3190884Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1)
2022-11-23T02:59:37.3269468Z Requirement already satisfied: MarkupSafe>=2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from jinja2->-r requirements.txt (line 17)) (2.1.1)
2022-11-23T02:59:37.3945835Z + python3 -m pip install boto3==1.19.12
2022-11-23T02:59:37.6963257Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T02:59:37.7197209Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12)
2022-11-23T02:59:37.7269720Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2)
2022-11-23T02:59:37.7309116Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2022-11-23T02:59:37.7327469Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12)
2022-11-23T02:59:37.7397848Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2022-11-23T02:59:37.7427532Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.12)
2022-11-23T02:59:37.7656021Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-11-23T02:59:38.0210857Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-11-23T02:59:54.2023831Z [scribe] Scribe access token not provided, sending report via boto3...
2022-11-23T02:59:54.2024195Z 
2022-11-23T02:59:54.2024581Z ----- Historic stats comparison result ------
2022-11-23T02:59:54.2024833Z 
2022-11-23T02:59:54.2025109Z     job: linux-bionic-cuda11.7-py3.10-gcc7
2022-11-23T02:59:54.2025858Z     commit: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T02:59:54.2026113Z 
2022-11-23T02:59:54.2026341Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-11-23T02:59:54.2026637Z 
2022-11-23T02:59:54.2026776Z     : (master)
2022-11-23T02:59:54.2027070Z     |
2022-11-23T02:59:54.2027432Z     * 1cfd3858ac (HEAD)              total time  2991.33s
2022-11-23T02:59:54.2030522Z     * 26322544b8 (base)  13 reports, total time  4193.12s ± 2023.52s
2022-11-23T02:59:54.2031080Z     * 7f4b4d2827         13 reports, total time  4157.15s ± 1970.21s
2022-11-23T02:59:54.2031580Z     * b50699f247         13 reports, total time  4190.33s ± 2008.01s
2022-11-23T02:59:54.2032065Z     * 8bf8e4d71e         13 reports, total time  4176.50s ± 2000.30s
2022-11-23T02:59:54.2032570Z     * ce856cee7e         12 reports, total time  3997.88s ± 1945.32s
2022-11-23T02:59:54.2033072Z     * 391b593ca2         13 reports, total time  4190.46s ± 2006.12s
2022-11-23T02:59:54.2033627Z     * 5bba783d21         13 reports, total time  4212.68s ± 2027.67s
2022-11-23T02:59:54.2034121Z     * ea920a1115         13 reports, total time  4210.45s ± 1980.15s
2022-11-23T02:59:54.2034799Z     * 74e62a1fef         13 reports, total time  4199.06s ± 2006.95s
2022-11-23T02:59:54.2035294Z     * 00b7d8ef23         13 reports, total time  4196.42s ± 2038.18s
2022-11-23T02:59:54.2035622Z     |
2022-11-23T02:59:54.2035908Z     :
2022-11-23T02:59:54.2036078Z 
2022-11-23T02:59:54.2036284Z Removed  (across  892 suites)     0 tests, totaling     0.00s
2022-11-23T02:59:54.2036697Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-11-23T02:59:54.2037123Z Added    (across   84 suites)   884 tests, totaling +3988.88s
2022-11-23T02:59:54.2665669Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2022-11-23T02:59:54.2666078Z with:
2022-11-23T02:59:54.2666359Z env:
2022-11-23T02:59:54.2666674Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:54.2667003Z   GPU_FLAG: --gpus all
2022-11-23T02:59:54.2667468Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:54.2667911Z ##[endgroup]
2022-11-23T02:59:54.2687762Z ##[group]Run set -eou pipefail
2022-11-23T02:59:54.2688166Z set -eou pipefail
2022-11-23T02:59:54.2688480Z 
2022-11-23T02:59:54.2688877Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2022-11-23T02:59:54.2689272Z for _ in $(seq 1440); do
2022-11-23T02:59:54.2689642Z     # Break if no ssh session exists anymore
2022-11-23T02:59:54.2690009Z     if [ "$(who)" = "" ]; then
2022-11-23T02:59:54.2690322Z       break
2022-11-23T02:59:54.2690666Z     fi
2022-11-23T02:59:54.2690979Z     echo "."
2022-11-23T02:59:54.2691279Z     sleep 5
2022-11-23T02:59:54.2691586Z done
2022-11-23T02:59:54.2704780Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:59:54.2705143Z env:
2022-11-23T02:59:54.2705458Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:54.2705786Z   GPU_FLAG: --gpus all
2022-11-23T02:59:54.2706250Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:54.2706706Z ##[endgroup]
2022-11-23T02:59:54.2737128Z Holding runner for 2 hours until all ssh sessions have logged out
2022-11-23T02:59:54.2786426Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-11-23T02:59:54.2786918Z # ignore expansion of "docker ps -q" since it could be empty
2022-11-23T02:59:54.2787321Z # shellcheck disable=SC2046
2022-11-23T02:59:54.2787687Z docker stop $(docker ps -q) || true
2022-11-23T02:59:54.2788049Z # Prune all of the docker images
2022-11-23T02:59:54.2788408Z docker system prune -af
2022-11-23T02:59:54.2800505Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T02:59:54.2800840Z env:
2022-11-23T02:59:54.2801141Z   GIT_DEFAULT_BRANCH: master
2022-11-23T02:59:54.2801469Z   GPU_FLAG: --gpus all
2022-11-23T02:59:54.2801904Z   DOCKER_CONTAINER_ID: 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:54.2802363Z ##[endgroup]
2022-11-23T02:59:54.7368393Z 2ed9f7ebac38
2022-11-23T02:59:55.8360569Z Deleted Containers:
2022-11-23T02:59:55.8361081Z 2ed9f7ebac3852afaaca9decf398bdfab892048ee1113bb973c15bd9aad5c82f
2022-11-23T02:59:55.8361372Z 
2022-11-23T03:00:00.8823290Z Deleted Images:
2022-11-23T03:00:00.8824254Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T03:00:00.8825400Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7@sha256:a44ece0129de4f14f08fcb1423a34c97f2f88e2969bfc5ad6f33b15b4dfcfea3
2022-11-23T03:00:00.8826139Z deleted: sha256:ca0e26c7bac33dbcc7a8dbe19f76edee4b91691e765d908dd755388b14acb91e
2022-11-23T03:00:00.8826668Z deleted: sha256:acbff221174f85354b2aba183449be48b8a649044c4d855f47ffe7c15f2f0593
2022-11-23T03:00:00.8827247Z deleted: sha256:6e2b2e86bf8d35dfec9e9487ce66c8ec4c5e366546510dbf2061107259075ec2
2022-11-23T03:00:00.8827764Z deleted: sha256:7ee6329a87fae79992a26743847bbc86bacc2ca03801efcdce7754a05ccb441f
2022-11-23T03:00:00.8828532Z deleted: sha256:6b5c166d5b90baf854740fd230f7d138850141e8dd1069c75900aa09e5e0dc90
2022-11-23T03:00:00.8829069Z deleted: sha256:ab05e7d25d0b30fc6f3cbf720ae9408ac11bcbdfea59731ae72dcc8aa3e520d6
2022-11-23T03:00:00.8829615Z deleted: sha256:f6246cb9656221ce7e6751e93b05e13424fb2583c6137d4edbfef96ceb0fbf4f
2022-11-23T03:00:00.8830148Z deleted: sha256:04d30f9a047eb1f0199adf2b2ca2512f9554b7caad28a2fb14a22a388d305a98
2022-11-23T03:00:00.8830765Z deleted: sha256:5c6550a005fd1407aa8a10de1057e8212385b7ef81fdb403eac9cb25b2cdd30a
2022-11-23T03:00:00.8831317Z deleted: sha256:2db874a48ccf9cf10f3600e1e7e0c5165c462d689f394d7be669d852c0f28f8e
2022-11-23T03:00:00.8831851Z deleted: sha256:1cb9632811ae406a7896c8ad49717c265bba77d94df3a67c23bd0cc1222c93fe
2022-11-23T03:00:00.8832372Z deleted: sha256:0b6a063f7a058d0226b7f958b16f7f2e4b68f39e6275890c7a7a0d339ebf9a70
2022-11-23T03:00:00.8832883Z deleted: sha256:afef4e32922b9a5b37e91ef12e33891a7d22f056ccb366412c89d8b775231c67
2022-11-23T03:00:00.8833432Z deleted: sha256:13dcda7fb091c2ea82bde2883af19edb94358834314a6d7d89de972fddf854cc
2022-11-23T03:00:00.8833993Z deleted: sha256:ba4023bbeb7ba3bdfc8353b5e973c00f1dae850fe0e4232344bca0d7bdc5ed21
2022-11-23T03:00:00.8834513Z deleted: sha256:eda87a7e3963049f09510a9664306ce96a1e843bd2ed9306aad33978f2415934
2022-11-23T03:00:00.8835037Z deleted: sha256:ad917be56ac1ba848d966f7332b79ef401b9420b7952fba45570faf98875de2c
2022-11-23T03:00:00.8835563Z deleted: sha256:3c2ba68cc2c00d49222172449200284f1caf0e9b25d7eeba6698297ef42c7059
2022-11-23T03:00:00.8836096Z deleted: sha256:1505992ac44d07e9564edf19e4ed03a31f7609cbfd0cc4bd607a988bd64d6592
2022-11-23T03:00:00.8836612Z deleted: sha256:7e3ce05cb4796e4f977d34b2c9499faabd362b94750f42ea36023785fd07210b
2022-11-23T03:00:00.8837134Z deleted: sha256:99012b24a24f7cbe1787c76d263d7cf9556d2542c33e1f4fde47d5a9bfced7bf
2022-11-23T03:00:00.8837655Z deleted: sha256:44a93e00ff78481f1922fd26545eba58d27c25ede11210755a18ac1089d34491
2022-11-23T03:00:00.8838154Z deleted: sha256:3d35c57c87aa053605d9839c481d2505ef05d74269206229cd53cf56291a84bc
2022-11-23T03:00:00.8838677Z deleted: sha256:d0644c40eee52273b45cfd6a9d443606fd3b1d570bba60df414bef688531865f
2022-11-23T03:00:00.8839221Z deleted: sha256:87f1587cf0fc998c2fd781ff08bd41ecd3f1dce58e89485eae3f97cf5e65a224
2022-11-23T03:00:00.8839764Z deleted: sha256:e967cbd5936283cfefeee219340c1dc42b04c590734c01da262f3591426d7033
2022-11-23T03:00:00.8840284Z deleted: sha256:cb4e3fb37d95ff54c5467335caa5a165d6951a524c72098c8aa50e91489a1cf7
2022-11-23T03:00:00.8840822Z deleted: sha256:c999a26e008b2a5a646e886f297ad5b2efd1ffc6de561d309eecf7e904d80411
2022-11-23T03:00:00.8841365Z deleted: sha256:0f7c27db48adc87df4d2d2275a157eb1b46971b7b9901cf97c7e5f95a0bd9f3a
2022-11-23T03:00:00.8841885Z deleted: sha256:6172bd74910d9d7ff816aac3ed295122ccc70f926d75b17196d3a8a7da581a48
2022-11-23T03:00:00.8842434Z deleted: sha256:f6c04aacc51d4155e8c2e978c915ad2c1a0d170bf3e933aeeaa66f7f5e9f529b
2022-11-23T03:00:00.8842981Z deleted: sha256:1a123d1e783cb340d39fb0fd6af62f0f31ec366902386b4843d78aa15341ca35
2022-11-23T03:00:00.8843500Z deleted: sha256:14682bd215f05536bf4a50be70610b9092a6f0c799b9adfa102ff191b669013f
2022-11-23T03:00:00.8843994Z deleted: sha256:055f5d43f9fee2ec66f09539d8c1624084e8029d0e74981a53354b7ff44c0af1
2022-11-23T03:00:00.8844528Z deleted: sha256:e1d86ed5bd48d6e66387d31531db825bf43e642ec87d3aea0ae602bda9362204
2022-11-23T03:00:00.8845065Z deleted: sha256:717c3163e25327a177ddf78de4dcdea20521942a9cb18c8e1b21e8336708455b
2022-11-23T03:00:00.8845585Z deleted: sha256:eb3a046c873cc3023f10cc7a3da759dda1fc3977511c26c8fe12d02cd35d9d3b
2022-11-23T03:00:00.8846400Z deleted: sha256:f7d654b861e155438fb4017f993e78f94e4ab971ba70a9923b9578da6f521079
2022-11-23T03:00:00.8846923Z deleted: sha256:9f6e64fe4d84e561aa5f4fb6ca895b16073451674b2e9234678faa71885b7c1a
2022-11-23T03:00:00.8847459Z deleted: sha256:cd4ed903887c2663e7ae16a705cdb277bfac87a35e8b3930e0f7c28f392243ff
2022-11-23T03:00:00.8848002Z deleted: sha256:6acd45a2d76545be71b8be690cb8bf0530cd1322d5aeb90ccd0025b76aeeb9be
2022-11-23T03:00:00.8848647Z deleted: sha256:5f6fad27350d1485f97817591e1e38f341c4f2466973c5afef012cde54640119
2022-11-23T03:00:00.8849164Z deleted: sha256:e77f111f1f3e2f93994c6b428a64619af7dad3012f71096de895d7965c6f6f54
2022-11-23T03:00:00.8849706Z deleted: sha256:1f60aaab75e1b24ad8a6dda4c7591a4cbe1c0233111faa95bbb121e71744f5e2
2022-11-23T03:00:00.8850233Z deleted: sha256:ed3557da864e7a48d82ca4fd51c078d177841f99d52a8480a8442fc6da95b3ef
2022-11-23T03:00:00.8850867Z deleted: sha256:e1cb5ac03c90992794bbcece5ef3a88b8d3dbfebb070b7e7b7dc6e1066a0bd0b
2022-11-23T03:00:00.8851415Z deleted: sha256:92b610b902f3255c897a349782a959cd391014cb7949989ae136548de42631f8
2022-11-23T03:00:00.8851906Z deleted: sha256:c52692915b570d66df58947dd8d7cba68fcfcf8fc296a447481578eb25879692
2022-11-23T03:00:00.8852436Z deleted: sha256:8d99114ec4c3615c49f00fa0fa40e0db8d9445f641c42ffdc127590c6fa9fe9a
2022-11-23T03:00:00.8852969Z deleted: sha256:b3b2969d5ff622447527c54efa6f9d15f8bc70de5db17f46ae11efd709999fbd
2022-11-23T03:00:00.8853499Z deleted: sha256:bfb13edd0300dc2515d887adb670832f4a85848e65db3348bbb1fd74e90fc5fc
2022-11-23T03:00:00.8854050Z deleted: sha256:c1ca91bea3c8e97f0d48fbfe60f987231f95874fcf29a94b76bb67dca54cad18
2022-11-23T03:00:00.8854588Z deleted: sha256:b5ef1223e23564cc06af079be55d6a3962040fb669c5d8724912ff6ea2075e59
2022-11-23T03:00:00.8855114Z deleted: sha256:69f57fbceb1b420d7e4697e0f6514887b0805ee0059bea7d51e0a832962e74bf
2022-11-23T03:00:00.8855414Z 
2022-11-23T03:00:00.8855586Z Total reclaimed space: 20.08GB
2022-11-23T03:00:00.8962236Z Post job cleanup.
2022-11-23T03:00:00.9003475Z Post job cleanup.
2022-11-23T03:00:01.0397244Z [command]/usr/bin/git version
2022-11-23T03:00:01.0453373Z git version 2.37.1
2022-11-23T03:00:01.0518618Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/adc02783-f095-4303-8370-a9409e950f00' before making global git config changes
2022-11-23T03:00:01.0519274Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T03:00:01.0525453Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T03:00:01.0570370Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T03:00:01.0607242Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T03:00:01.0932459Z Entering 'android/libs/fbjni'
2022-11-23T03:00:01.0978575Z Entering 'third_party/FP16'
2022-11-23T03:00:01.1020964Z Entering 'third_party/FXdiv'
2022-11-23T03:00:01.1062143Z Entering 'third_party/NNPACK'
2022-11-23T03:00:01.1104808Z Entering 'third_party/QNNPACK'
2022-11-23T03:00:01.1147860Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T03:00:01.1191438Z Entering 'third_party/XNNPACK'
2022-11-23T03:00:01.1245954Z Entering 'third_party/benchmark'
2022-11-23T03:00:01.1288490Z Entering 'third_party/cpuinfo'
2022-11-23T03:00:01.1331001Z Entering 'third_party/cub'
2022-11-23T03:00:01.1373504Z Entering 'third_party/cudnn_frontend'
2022-11-23T03:00:01.1421514Z Entering 'third_party/cutlass'
2022-11-23T03:00:01.1471353Z Entering 'third_party/eigen'
2022-11-23T03:00:01.1517033Z Entering 'third_party/fbgemm'
2022-11-23T03:00:01.1559200Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T03:00:01.1601295Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T03:00:01.1643445Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T03:00:01.1684404Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T03:00:01.1727485Z Entering 'third_party/flatbuffers'
2022-11-23T03:00:01.1771755Z Entering 'third_party/fmt'
2022-11-23T03:00:01.1815476Z Entering 'third_party/foxi'
2022-11-23T03:00:01.1856820Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T03:00:01.1898058Z Entering 'third_party/gloo'
2022-11-23T03:00:01.1940142Z Entering 'third_party/googletest'
2022-11-23T03:00:01.1981910Z Entering 'third_party/ideep'
2022-11-23T03:00:01.2022943Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T03:00:01.2065493Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T03:00:01.2114143Z Entering 'third_party/ios-cmake'
2022-11-23T03:00:01.2156503Z Entering 'third_party/ittapi'
2022-11-23T03:00:01.2197967Z Entering 'third_party/kineto'
2022-11-23T03:00:01.2239915Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T03:00:01.2281550Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T03:00:01.2324491Z Entering 'third_party/nccl/nccl'
2022-11-23T03:00:01.2366664Z Entering 'third_party/neon2sse'
2022-11-23T03:00:01.2409731Z Entering 'third_party/nlohmann'
2022-11-23T03:00:01.2454430Z Entering 'third_party/onnx'
2022-11-23T03:00:01.2511524Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T03:00:01.2554110Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T03:00:01.2598488Z Entering 'third_party/onnx-tensorrt'
2022-11-23T03:00:01.2639465Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T03:00:01.2685717Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T03:00:01.2728228Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T03:00:01.2770347Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T03:00:01.2815923Z Entering 'third_party/pocketfft'
2022-11-23T03:00:01.2859235Z Entering 'third_party/protobuf'
2022-11-23T03:00:01.2904058Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T03:00:01.2945413Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T03:00:01.2988794Z Entering 'third_party/psimd'
2022-11-23T03:00:01.3031215Z Entering 'third_party/pthreadpool'
2022-11-23T03:00:01.3073278Z Entering 'third_party/pybind11'
2022-11-23T03:00:01.3115614Z Entering 'third_party/python-enum'
2022-11-23T03:00:01.3158201Z Entering 'third_party/python-peachpy'
2022-11-23T03:00:01.3200432Z Entering 'third_party/python-six'
2022-11-23T03:00:01.3241788Z Entering 'third_party/sleef'
2022-11-23T03:00:01.3283515Z Entering 'third_party/tbb'
2022-11-23T03:00:01.3327343Z Entering 'third_party/tensorpipe'
2022-11-23T03:00:01.3369385Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T03:00:01.3410263Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T03:00:01.3453463Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T03:00:01.3494094Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T03:00:01.3535471Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T03:00:01.3579539Z Entering 'third_party/zstd'
2022-11-23T03:00:01.3637302Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T03:00:01.3667121Z http.https://github.com/.extraheader
2022-11-23T03:00:01.3676454Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-11-23T03:00:01.3712053Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T03:00:01.4031633Z Entering 'android/libs/fbjni'
2022-11-23T03:00:01.4056391Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4090093Z Entering 'third_party/FP16'
2022-11-23T03:00:01.4115031Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4148649Z Entering 'third_party/FXdiv'
2022-11-23T03:00:01.4174604Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4206180Z Entering 'third_party/NNPACK'
2022-11-23T03:00:01.4231533Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4263732Z Entering 'third_party/QNNPACK'
2022-11-23T03:00:01.4288748Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4321030Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T03:00:01.4346481Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4379167Z Entering 'third_party/XNNPACK'
2022-11-23T03:00:01.4403568Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4448976Z Entering 'third_party/benchmark'
2022-11-23T03:00:01.4473293Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4505691Z Entering 'third_party/cpuinfo'
2022-11-23T03:00:01.4531235Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4563651Z Entering 'third_party/cub'
2022-11-23T03:00:01.4589359Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4622310Z Entering 'third_party/cudnn_frontend'
2022-11-23T03:00:01.4647062Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4686321Z Entering 'third_party/cutlass'
2022-11-23T03:00:01.4711097Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4752046Z Entering 'third_party/eigen'
2022-11-23T03:00:01.4776747Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4812237Z Entering 'third_party/fbgemm'
2022-11-23T03:00:01.4836858Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4870962Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T03:00:01.4896488Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4929570Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T03:00:01.4955035Z http.https://github.com/.extraheader
2022-11-23T03:00:01.4988480Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T03:00:01.5012908Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5044951Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T03:00:01.5070402Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5104690Z Entering 'third_party/flatbuffers'
2022-11-23T03:00:01.5130230Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5164513Z Entering 'third_party/fmt'
2022-11-23T03:00:01.5190024Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5223911Z Entering 'third_party/foxi'
2022-11-23T03:00:01.5250129Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5282108Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T03:00:01.5307513Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5341267Z Entering 'third_party/gloo'
2022-11-23T03:00:01.5365635Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5398618Z Entering 'third_party/googletest'
2022-11-23T03:00:01.5425206Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5457774Z Entering 'third_party/ideep'
2022-11-23T03:00:01.5483113Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5516153Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T03:00:01.5541051Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5576276Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T03:00:01.5601824Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5642833Z Entering 'third_party/ios-cmake'
2022-11-23T03:00:01.5668686Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5701555Z Entering 'third_party/ittapi'
2022-11-23T03:00:01.5726645Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5759682Z Entering 'third_party/kineto'
2022-11-23T03:00:01.5784578Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5818677Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T03:00:01.5843498Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5876790Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T03:00:01.5901764Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5937038Z Entering 'third_party/nccl/nccl'
2022-11-23T03:00:01.5961926Z http.https://github.com/.extraheader
2022-11-23T03:00:01.5995587Z Entering 'third_party/neon2sse'
2022-11-23T03:00:01.6021060Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6053655Z Entering 'third_party/nlohmann'
2022-11-23T03:00:01.6078827Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6113954Z Entering 'third_party/onnx'
2022-11-23T03:00:01.6139383Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6186842Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T03:00:01.6211964Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6244860Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T03:00:01.6271211Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6308287Z Entering 'third_party/onnx-tensorrt'
2022-11-23T03:00:01.6335761Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6368388Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T03:00:01.6393458Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6433670Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T03:00:01.6458919Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6493795Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T03:00:01.6518132Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6552820Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T03:00:01.6578795Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6618186Z Entering 'third_party/pocketfft'
2022-11-23T03:00:01.6643760Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6677642Z Entering 'third_party/protobuf'
2022-11-23T03:00:01.6703635Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6741826Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T03:00:01.6766567Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6799791Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T03:00:01.6824390Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6859805Z Entering 'third_party/psimd'
2022-11-23T03:00:01.6884879Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6918316Z Entering 'third_party/pthreadpool'
2022-11-23T03:00:01.6944205Z http.https://github.com/.extraheader
2022-11-23T03:00:01.6976944Z Entering 'third_party/pybind11'
2022-11-23T03:00:01.7002212Z http.https://github.com/.extraheader
2022-11-23T03:00:01.7035327Z Entering 'third_party/python-enum'
2022-11-23T03:00:01.7060275Z http.https://github.com/.extraheader
2022-11-23T03:00:01.7092752Z Entering 'third_party/python-peachpy'
2022-11-23T03:00:01.7117264Z http.https://github.com/.extraheader
2022-11-23T03:00:01.7151040Z Entering 'third_party/python-six'
2022-11-23T03:00:01.7177511Z http.https://github.com/.extraheader
2022-11-23T03:00:01.7210269Z Entering 'third_party/sleef'
2022-11-23T03:00:01.7234610Z http.https://github.com/.extraheader
2022-11-23T03:00:01.7267789Z Entering 'third_party/tbb'
2022-11-23T03:00:01.7293286Z http.https://github.com/.extraheader
2022-11-23T03:00:01.7328054Z Entering 'third_party/tensorpipe'
2022-11-23T03:00:01.7352360Z http.https://github.com/.extraheader
2022-11-23T03:00:01.7385464Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T03:00:01.7409786Z http.https://github.com/.extraheader
2022-11-23T03:00:01.7442959Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T03:00:01.7467094Z http.https://github.com/.extraheader
2022-11-23T03:00:01.7500685Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T03:00:01.7524360Z http.https://github.com/.extraheader
2022-11-23T03:00:01.7557596Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T03:00:01.7582469Z http.https://github.com/.extraheader
2022-11-23T03:00:01.7615502Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T03:00:01.7640171Z http.https://github.com/.extraheader
2022-11-23T03:00:01.7676537Z Entering 'third_party/zstd'
2022-11-23T03:00:01.7702212Z http.https://github.com/.extraheader
2022-11-23T03:00:01.8064767Z Cleaning up orphan processes