2022-11-23T01:10:29.4158197Z Requested labels: linux.8xlarge.nvidia.gpu
2022-11-23T01:10:29.4158313Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/master
2022-11-23T01:10:29.4158355Z Waiting for a runner to pick up this job...
2022-11-23T01:10:29.5635288Z Job is about to start running on the runner: i-099ed1720beaf2723 (organization)
2022-11-23T01:10:33.6612775Z Current runner version: '2.299.1'
2022-11-23T01:10:33.6620202Z Runner name: 'i-099ed1720beaf2723'
2022-11-23T01:10:33.6620863Z Runner group name: 'Default'
2022-11-23T01:10:33.6621618Z Machine name: 'ip-10-0-2-220'
2022-11-23T01:10:33.6624525Z ##[group]GITHUB_TOKEN Permissions
2022-11-23T01:10:33.6625418Z Actions: write
2022-11-23T01:10:33.6625858Z Checks: write
2022-11-23T01:10:33.6626294Z Contents: write
2022-11-23T01:10:33.6626735Z Deployments: write
2022-11-23T01:10:33.6627532Z Discussions: write
2022-11-23T01:10:33.6627969Z Issues: write
2022-11-23T01:10:33.6628334Z Metadata: read
2022-11-23T01:10:33.6628789Z Packages: write
2022-11-23T01:10:33.6629279Z Pages: write
2022-11-23T01:10:33.6629681Z PullRequests: write
2022-11-23T01:10:33.6630179Z RepositoryProjects: write
2022-11-23T01:10:33.6630664Z SecurityEvents: write
2022-11-23T01:10:33.6631053Z Statuses: write
2022-11-23T01:10:33.6631528Z ##[endgroup]
2022-11-23T01:10:33.6635711Z Secret source: Actions
2022-11-23T01:10:33.6636526Z Prepare workflow directory
2022-11-23T01:10:33.7935741Z Prepare all required actions
2022-11-23T01:10:33.8159046Z Getting action download info
2022-11-23T01:10:34.0119146Z Download action repository 'pytorch/test-infra@main' (SHA:c57ff4d9a93667a5571a80a0e92c3e2674aeedfd)
2022-11-23T01:10:34.3211275Z Download action repository 'pytorch/pytorch@master' (SHA:1cfd3858ac54fe3883534309081631a0a892ba3f)
2022-11-23T01:10:37.5242877Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2022-11-23T01:10:37.8057709Z Getting action download info
2022-11-23T01:10:37.9539941Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2022-11-23T01:10:38.1244066Z Getting action download info
2022-11-23T01:10:38.2675180Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767)
2022-11-23T01:10:38.4382022Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2022-11-23T01:10:38.4384676Z ##[group] Inputs
2022-11-23T01:10:38.4385084Z   build-environment: linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T01:10:38.4386347Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 2, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 3, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 4, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "distributed", shard: 1, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 2, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 3, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "deploy", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
]}

2022-11-23T01:10:38.4388163Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:10:38.4388662Z   sync-tag: 
2022-11-23T01:10:38.4389816Z   timeout-minutes: 240
2022-11-23T01:10:38.4390098Z ##[endgroup]
2022-11-23T01:10:38.4390962Z Complete job name: linux-bionic-cuda11.6-py3.10-gcc7 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu)
2022-11-23T01:10:38.5477549Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2022-11-23T01:10:38.5477935Z with:
2022-11-23T01:10:38.5478501Z   github-secret: ***
2022-11-23T01:10:38.5478796Z   activate-with-label: false
2022-11-23T01:10:38.5479068Z   label: with-ssh
2022-11-23T01:10:38.5479314Z   remove-existing-keys: true
2022-11-23T01:10:38.5479573Z env:
2022-11-23T01:10:38.5479814Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:38.5480059Z ##[endgroup]
2022-11-23T01:10:38.6545949Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2022-11-23T01:10:38.6777781Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-11-23T01:10:38.6778160Z with:
2022-11-23T01:10:38.6778381Z   submodules: recursive
2022-11-23T01:10:38.6778637Z   fetch-depth: 0
2022-11-23T01:10:38.6778864Z env:
2022-11-23T01:10:38.6779098Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:38.6779421Z ##[endgroup]
2022-11-23T01:10:38.7063784Z ##[group]Run retry () {
2022-11-23T01:10:38.7064116Z retry () {
2022-11-23T01:10:38.7064408Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-11-23T01:10:38.7064708Z }
2022-11-23T01:10:38.7064966Z echo "${GITHUB_WORKSPACE}"
2022-11-23T01:10:38.7065247Z if [ -z "${NO_SUDO}" ]; then
2022-11-23T01:10:38.7065558Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T01:10:38.7065838Z else
2022-11-23T01:10:38.7066093Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-11-23T01:10:38.7066360Z fi
2022-11-23T01:10:38.7066652Z mkdir "${GITHUB_WORKSPACE}"
2022-11-23T01:10:38.7084895Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:10:38.7085211Z env:
2022-11-23T01:10:38.7085459Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:38.7085715Z   NO_SUDO: 
2022-11-23T01:10:38.7085932Z ##[endgroup]
2022-11-23T01:10:38.7210849Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:10:38.7611142Z ##[group]Run malfet/checkout@silent-checkout
2022-11-23T01:10:38.7611443Z with:
2022-11-23T01:10:38.7611715Z   ref: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:10:38.7611980Z   fetch-depth: 0
2022-11-23T01:10:38.7612234Z   submodules: recursive
2022-11-23T01:10:38.7612495Z   quiet-checkout: true
2022-11-23T01:10:38.7612753Z   repository: pytorch/pytorch
2022-11-23T01:10:38.7613209Z   token: ***
2022-11-23T01:10:38.7613456Z   ssh-strict: true
2022-11-23T01:10:38.7613707Z   persist-credentials: true
2022-11-23T01:10:38.7613969Z   clean: true
2022-11-23T01:10:38.7614206Z   lfs: false
2022-11-23T01:10:38.7614451Z   set-safe-directory: true
2022-11-23T01:10:38.7614699Z env:
2022-11-23T01:10:38.7614937Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:10:38.7615179Z ##[endgroup]
2022-11-23T01:10:38.9089676Z Syncing repository: pytorch/pytorch
2022-11-23T01:10:38.9091600Z ##[group]Getting Git version info
2022-11-23T01:10:38.9092157Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T01:10:38.9092751Z [command]/usr/bin/git version
2022-11-23T01:10:38.9093023Z git version 2.37.1
2022-11-23T01:10:38.9097518Z ##[endgroup]
2022-11-23T01:10:38.9117045Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/5bff68eb-b072-4d9b-afb0-724852b04e99' before making global git config changes
2022-11-23T01:10:38.9118337Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T01:10:38.9124410Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:10:38.9168764Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-11-23T01:10:38.9175349Z ##[group]Initializing the repository
2022-11-23T01:10:38.9179037Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:10:38.9210654Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-11-23T01:10:38.9211332Z hint: is subject to change. To configure the initial branch name to use in all
2022-11-23T01:10:38.9211913Z hint: of your new repositories, which will suppress this warning, call:
2022-11-23T01:10:38.9212234Z hint: 
2022-11-23T01:10:38.9212601Z hint: 	git config --global init.defaultBranch <name>
2022-11-23T01:10:38.9212898Z hint: 
2022-11-23T01:10:38.9213287Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-11-23T01:10:38.9213783Z hint: 'development'. The just-created branch can be renamed via this command:
2022-11-23T01:10:38.9214109Z hint: 
2022-11-23T01:10:38.9214559Z hint: 	git branch -m <name>
2022-11-23T01:10:38.9215172Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-11-23T01:10:38.9224528Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-11-23T01:10:38.9257125Z ##[endgroup]
2022-11-23T01:10:38.9257612Z ##[group]Disabling automatic garbage collection
2022-11-23T01:10:38.9261590Z [command]/usr/bin/git config --local gc.auto 0
2022-11-23T01:10:38.9291544Z ##[endgroup]
2022-11-23T01:10:38.9292477Z ##[group]Setting up auth
2022-11-23T01:10:38.9301440Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T01:10:38.9334361Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T01:10:38.9671074Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T01:10:38.9700786Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T01:10:38.9992676Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T01:10:39.0042764Z ##[endgroup]
2022-11-23T01:10:39.0043237Z ##[group]Fetching the repository
2022-11-23T01:10:39.0051446Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-11-23T01:11:30.8824132Z [command]/usr/bin/git rev-parse --verify --quiet 1cfd3858ac54fe3883534309081631a0a892ba3f^{object}
2022-11-23T01:11:30.8852337Z 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:11:30.8859910Z ##[endgroup]
2022-11-23T01:11:30.8860484Z ##[group]Determining the checkout info
2022-11-23T01:11:30.8860929Z ##[endgroup]
2022-11-23T01:11:30.8861388Z ##[group]Checking out the ref
2022-11-23T01:11:30.8865672Z [command]/usr/bin/git checkout --quiet --force 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:11:32.6108001Z ##[endgroup]
2022-11-23T01:11:32.6108731Z ##[group]Setting up auth for fetching submodules
2022-11-23T01:11:32.6115002Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-11-23T01:11:32.6171918Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-11-23T01:11:32.6203927Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-11-23T01:11:32.6235328Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-11-23T01:11:32.6264264Z ##[endgroup]
2022-11-23T01:11:32.6264704Z ##[group]Fetching submodules
2022-11-23T01:11:32.6270234Z [command]/usr/bin/git submodule sync --recursive
2022-11-23T01:11:32.6589827Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-11-23T01:11:32.6893168Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-11-23T01:11:32.6895941Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-11-23T01:11:32.6898945Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-11-23T01:11:32.6902316Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-11-23T01:11:32.6905649Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-11-23T01:11:32.6909949Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2022-11-23T01:11:32.6913774Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-11-23T01:11:32.6917442Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-11-23T01:11:32.6921402Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-11-23T01:11:32.6925484Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-11-23T01:11:32.6929727Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-11-23T01:11:32.6933990Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2022-11-23T01:11:32.6938552Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-11-23T01:11:32.6943115Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-11-23T01:11:32.6947952Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-11-23T01:11:32.6953089Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-11-23T01:11:32.6958232Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-11-23T01:11:32.6963274Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:11:32.6968753Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-11-23T01:11:32.6974058Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-11-23T01:11:32.6980185Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-11-23T01:11:32.6985849Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-11-23T01:11:32.6993777Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-11-23T01:11:32.6999443Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-11-23T01:11:32.7005564Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-11-23T01:11:32.7011795Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-11-23T01:11:32.7017902Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-11-23T01:11:32.7024231Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-11-23T01:11:32.7031542Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-11-23T01:11:32.7038148Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-11-23T01:11:32.7044972Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-11-23T01:11:32.7051830Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-11-23T01:11:32.7058987Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-11-23T01:11:32.7066103Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-11-23T01:11:32.7074272Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-11-23T01:11:32.7081462Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-11-23T01:11:32.7088961Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-11-23T01:11:32.7096560Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-11-23T01:11:32.7104409Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-11-23T01:11:32.7113155Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-11-23T01:11:32.7121261Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-11-23T01:11:32.7149387Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-11-23T01:11:33.0094597Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-11-23T01:11:33.2014378Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-11-23T01:11:33.4207446Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-11-23T01:11:33.7435943Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-11-23T01:11:34.0171466Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2022-11-23T01:11:36.0561878Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-11-23T01:11:41.6888376Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-11-23T01:11:42.1031435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-11-23T01:11:42.6812043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-11-23T01:11:44.2945918Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-11-23T01:11:45.7997703Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2022-11-23T01:11:47.5697451Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-11-23T01:11:54.7224149Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-11-23T01:11:55.4690435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-11-23T01:11:56.9828667Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-11-23T01:11:58.1509310Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-11-23T01:11:58.3511997Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-11-23T01:11:58.8384068Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-11-23T01:11:59.1539994Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-11-23T01:12:00.1613579Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-11-23T01:12:00.5921241Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-11-23T01:12:00.8026325Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-11-23T01:12:01.0915136Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-11-23T01:12:04.3687735Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-11-23T01:12:04.8778256Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-11-23T01:12:05.2785409Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-11-23T01:12:11.8033267Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-11-23T01:12:13.5065719Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-11-23T01:12:13.9787638Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-11-23T01:12:14.2367373Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-11-23T01:12:20.9946261Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-11-23T01:12:21.2258979Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-11-23T01:12:21.4339667Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-11-23T01:12:22.3287281Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-11-23T01:12:22.5463411Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-11-23T01:12:22.8939537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-11-23T01:12:23.1983717Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-11-23T01:12:23.7700250Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-11-23T01:12:26.3433388Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-11-23T01:12:26.8553283Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-11-23T01:12:29.2605821Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-11-23T01:12:29.2730997Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-11-23T01:12:29.2827600Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-11-23T01:12:29.3106431Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-11-23T01:12:29.3374792Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-11-23T01:12:29.3793770Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2022-11-23T01:12:30.1454605Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-11-23T01:12:30.1699406Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-11-23T01:12:30.2918986Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2022-11-23T01:12:30.3322416Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-11-23T01:12:30.6897719Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2022-11-23T01:12:31.2057586Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2022-11-23T01:12:31.5063681Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-11-23T01:12:31.5617488Z Submodule path 'third_party/fbgemm': checked out '4d1738b3142a6cb0c032cd639e239566010b054a'
2022-11-23T01:12:31.5635724Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:12:31.5638891Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:12:31.5642246Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:12:31.5645655Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:12:31.5672285Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-11-23T01:12:32.4859417Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-11-23T01:12:33.0554607Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-11-23T01:12:34.0485159Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-11-23T01:12:34.5723579Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-11-23T01:12:34.6939369Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-11-23T01:12:34.7636024Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-11-23T01:12:34.7748822Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-11-23T01:12:34.8889447Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-11-23T01:12:34.9305374Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b'
2022-11-23T01:12:34.9405043Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-11-23T01:12:34.9873748Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-11-23T01:12:35.0154806Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195'
2022-11-23T01:12:35.0696488Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-11-23T01:12:35.0829260Z Submodule path 'third_party/ideep': checked out '5ddc65efe0428bbce2942b3ce5e3ce15239abe2f'
2022-11-23T01:12:35.0846538Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-11-23T01:12:35.0874481Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-11-23T01:12:43.9040489Z Submodule path 'third_party/ideep/mkl-dnn': checked out 'd19d0f795c60695bd32f894c6f01771b2dfbe24d'
2022-11-23T01:12:43.9060574Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:12:43.9088539Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-11-23T01:12:53.0601965Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '650085b2f3643aad05c629425983491d63b5c289'
2022-11-23T01:12:53.0716352Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-11-23T01:12:53.0883850Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-11-23T01:12:53.1999653Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562'
2022-11-23T01:12:53.2016895Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:12:53.2020196Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:12:53.2046191Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-11-23T01:12:54.3493867Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-11-23T01:12:55.6331368Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-11-23T01:12:55.6971321Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-11-23T01:12:55.7211598Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2022-11-23T01:12:55.7366588Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-11-23T01:12:55.8679679Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-11-23T01:12:56.1838589Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-11-23T01:12:56.1870536Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-11-23T01:12:56.1873798Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-11-23T01:12:56.1900653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-11-23T01:12:56.5938790Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-11-23T01:12:57.4892285Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-11-23T01:12:57.5272596Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-11-23T01:12:57.5449014Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-11-23T01:12:57.5466303Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:12:57.5493160Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-11-23T01:12:59.4064320Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-11-23T01:12:59.4086205Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:12:59.4089313Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:12:59.4116620Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-11-23T01:12:59.8209479Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-11-23T01:13:01.8467328Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-11-23T01:13:01.9274005Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-11-23T01:13:01.9290892Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:01.9318193Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-11-23T01:13:02.9401521Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-11-23T01:13:02.9542988Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-11-23T01:13:03.2891956Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-11-23T01:13:03.2915066Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:03.2918532Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:03.2946029Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-11-23T01:13:03.7376674Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-11-23T01:13:04.7790924Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-11-23T01:13:04.8602668Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-11-23T01:13:04.8695253Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-11-23T01:13:04.8819673Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-11-23T01:13:04.9224860Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917'
2022-11-23T01:13:04.9366598Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-11-23T01:13:04.9661228Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-11-23T01:13:04.9766641Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-11-23T01:13:05.0289981Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-11-23T01:13:05.1669169Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-11-23T01:13:05.1977322Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-11-23T01:13:05.1995996Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:05.1999209Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:05.2002533Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:05.2006011Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:05.2033409Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-11-23T01:13:06.2265099Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-11-23T01:13:06.5137961Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-11-23T01:13:07.8840465Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-11-23T01:13:08.8487446Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-11-23T01:13:08.8658679Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-11-23T01:13:08.9455932Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-11-23T01:13:08.9784790Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-11-23T01:13:08.9801419Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:08.9828422Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-11-23T01:13:09.2109464Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-11-23T01:13:09.3727621Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-11-23T01:13:09.3768861Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-11-23T01:13:09.4091084Z Entering 'android/libs/fbjni'
2022-11-23T01:13:09.4134016Z Entering 'third_party/FP16'
2022-11-23T01:13:09.4178187Z Entering 'third_party/FXdiv'
2022-11-23T01:13:09.4221200Z Entering 'third_party/NNPACK'
2022-11-23T01:13:09.4265107Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:09.4308379Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:09.4350918Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:09.4404042Z Entering 'third_party/benchmark'
2022-11-23T01:13:09.4447343Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:09.4490763Z Entering 'third_party/cub'
2022-11-23T01:13:09.4532852Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:09.4580540Z Entering 'third_party/cutlass'
2022-11-23T01:13:09.4631441Z Entering 'third_party/eigen'
2022-11-23T01:13:09.4676120Z Entering 'third_party/fbgemm'
2022-11-23T01:13:09.4718172Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:09.4761655Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:09.4804264Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:09.4846235Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:09.4888516Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:09.4934120Z Entering 'third_party/fmt'
2022-11-23T01:13:09.4976149Z Entering 'third_party/foxi'
2022-11-23T01:13:09.5017839Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:09.5059627Z Entering 'third_party/gloo'
2022-11-23T01:13:09.5101116Z Entering 'third_party/googletest'
2022-11-23T01:13:09.5143105Z Entering 'third_party/ideep'
2022-11-23T01:13:09.5185746Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:09.5231397Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:09.5281026Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:09.5322472Z Entering 'third_party/ittapi'
2022-11-23T01:13:09.5364386Z Entering 'third_party/kineto'
2022-11-23T01:13:09.5406315Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:09.5448303Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:09.5491948Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:09.5533969Z Entering 'third_party/neon2sse'
2022-11-23T01:13:09.5575817Z Entering 'third_party/nlohmann'
2022-11-23T01:13:09.5619396Z Entering 'third_party/onnx'
2022-11-23T01:13:09.5674512Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:09.5715767Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:09.5760305Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:09.5800600Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:09.5846302Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:09.5887801Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:09.5929399Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:09.5975376Z Entering 'third_party/pocketfft'
2022-11-23T01:13:09.6016993Z Entering 'third_party/protobuf'
2022-11-23T01:13:09.6062936Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:09.6104214Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:09.6147379Z Entering 'third_party/psimd'
2022-11-23T01:13:09.6190287Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:09.6232212Z Entering 'third_party/pybind11'
2022-11-23T01:13:09.6275624Z Entering 'third_party/python-enum'
2022-11-23T01:13:09.6317156Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:09.6358839Z Entering 'third_party/python-six'
2022-11-23T01:13:09.6400830Z Entering 'third_party/sleef'
2022-11-23T01:13:09.6442617Z Entering 'third_party/tbb'
2022-11-23T01:13:09.6486304Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:09.6528309Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:09.6569372Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:09.6609617Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:09.6651075Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:09.6691707Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:09.6735361Z Entering 'third_party/zstd'
2022-11-23T01:13:09.6789494Z ##[endgroup]
2022-11-23T01:13:09.6790205Z ##[group]Persisting credentials for submodules
2022-11-23T01:13:09.6795845Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-11-23T01:13:09.7105031Z Entering 'android/libs/fbjni'
2022-11-23T01:13:09.7145913Z Entering 'third_party/FP16'
2022-11-23T01:13:09.7188230Z Entering 'third_party/FXdiv'
2022-11-23T01:13:09.7230480Z Entering 'third_party/NNPACK'
2022-11-23T01:13:09.7271581Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:09.7312707Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:09.7354163Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:09.7405279Z Entering 'third_party/benchmark'
2022-11-23T01:13:09.7446242Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:09.7488865Z Entering 'third_party/cub'
2022-11-23T01:13:09.7529895Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:09.7576384Z Entering 'third_party/cutlass'
2022-11-23T01:13:09.7625101Z Entering 'third_party/eigen'
2022-11-23T01:13:09.7669230Z Entering 'third_party/fbgemm'
2022-11-23T01:13:09.7709927Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:09.7750926Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:09.7793526Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:09.7835187Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:09.7877852Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:09.7920693Z Entering 'third_party/fmt'
2022-11-23T01:13:09.7962043Z Entering 'third_party/foxi'
2022-11-23T01:13:09.8002137Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:09.8042707Z Entering 'third_party/gloo'
2022-11-23T01:13:09.8083417Z Entering 'third_party/googletest'
2022-11-23T01:13:09.8124349Z Entering 'third_party/ideep'
2022-11-23T01:13:09.8165472Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:09.8207912Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:09.8256883Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:09.8299088Z Entering 'third_party/ittapi'
2022-11-23T01:13:09.8339830Z Entering 'third_party/kineto'
2022-11-23T01:13:09.8380184Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:09.8421232Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:09.8464775Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:09.8506101Z Entering 'third_party/neon2sse'
2022-11-23T01:13:09.8546261Z Entering 'third_party/nlohmann'
2022-11-23T01:13:09.8588767Z Entering 'third_party/onnx'
2022-11-23T01:13:09.8643169Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:09.8683723Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:09.8726982Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:09.8767641Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:09.8813933Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:09.8856631Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:09.8898144Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:09.8943934Z Entering 'third_party/pocketfft'
2022-11-23T01:13:09.8985802Z Entering 'third_party/protobuf'
2022-11-23T01:13:09.9030824Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:09.9071205Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:09.9113553Z Entering 'third_party/psimd'
2022-11-23T01:13:09.9154534Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:09.9194897Z Entering 'third_party/pybind11'
2022-11-23T01:13:09.9235952Z Entering 'third_party/python-enum'
2022-11-23T01:13:09.9276537Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:09.9317157Z Entering 'third_party/python-six'
2022-11-23T01:13:09.9358810Z Entering 'third_party/sleef'
2022-11-23T01:13:09.9399861Z Entering 'third_party/tbb'
2022-11-23T01:13:09.9442785Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:09.9484981Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:09.9525888Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:09.9566421Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:09.9607665Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:09.9647611Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:09.9690466Z Entering 'third_party/zstd'
2022-11-23T01:13:09.9744536Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-11-23T01:13:10.0051548Z Entering 'android/libs/fbjni'
2022-11-23T01:13:10.0091955Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-11-23T01:13:10.0109404Z Entering 'third_party/FP16'
2022-11-23T01:13:10.0148133Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-11-23T01:13:10.0164902Z Entering 'third_party/FXdiv'
2022-11-23T01:13:10.0205163Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-11-23T01:13:10.0222276Z Entering 'third_party/NNPACK'
2022-11-23T01:13:10.0262755Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-11-23T01:13:10.0281731Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:10.0321800Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-11-23T01:13:10.0338846Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:10.0377232Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2022-11-23T01:13:10.0395376Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:10.0434997Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-11-23T01:13:10.0463353Z Entering 'third_party/benchmark'
2022-11-23T01:13:10.0503636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:10.0521619Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:10.0560126Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-11-23T01:13:10.0577538Z Entering 'third_party/cub'
2022-11-23T01:13:10.0616666Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-11-23T01:13:10.0634179Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:10.0673032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-11-23T01:13:10.0695975Z Entering 'third_party/cutlass'
2022-11-23T01:13:10.0735801Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2022-11-23T01:13:10.0761286Z Entering 'third_party/eigen'
2022-11-23T01:13:10.0799704Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-11-23T01:13:10.0818968Z Entering 'third_party/fbgemm'
2022-11-23T01:13:10.0858598Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-11-23T01:13:10.0875834Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:10.0914001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-11-23T01:13:10.0930643Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:10.0969148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-11-23T01:13:10.0986433Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:10.1024025Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:10.1042148Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:10.1081263Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-11-23T01:13:10.1098802Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:10.1138062Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-11-23T01:13:10.1158735Z Entering 'third_party/fmt'
2022-11-23T01:13:10.1197385Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-11-23T01:13:10.1214079Z Entering 'third_party/foxi'
2022-11-23T01:13:10.1253826Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-11-23T01:13:10.1271222Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:10.1309638Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-11-23T01:13:10.1326905Z Entering 'third_party/gloo'
2022-11-23T01:13:10.1365536Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-11-23T01:13:10.1382603Z Entering 'third_party/googletest'
2022-11-23T01:13:10.1422152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:10.1439698Z Entering 'third_party/ideep'
2022-11-23T01:13:10.1477953Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-11-23T01:13:10.1494681Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:10.1533438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-11-23T01:13:10.1553443Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:10.1593118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-11-23T01:13:10.1617120Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:10.1656445Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-11-23T01:13:10.1673668Z Entering 'third_party/ittapi'
2022-11-23T01:13:10.1711731Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-11-23T01:13:10.1728683Z Entering 'third_party/kineto'
2022-11-23T01:13:10.1768695Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-11-23T01:13:10.1786113Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:10.1824363Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-11-23T01:13:10.1842413Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:10.1881384Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:10.1900464Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:10.1940284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-11-23T01:13:10.1958119Z Entering 'third_party/neon2sse'
2022-11-23T01:13:10.1996001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-11-23T01:13:10.2012746Z Entering 'third_party/nlohmann'
2022-11-23T01:13:10.2051398Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-11-23T01:13:10.2072248Z Entering 'third_party/onnx'
2022-11-23T01:13:10.2110597Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-11-23T01:13:10.2141708Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:10.2180886Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:10.2199123Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:10.2237888Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:10.2257353Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:10.2297593Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-11-23T01:13:10.2314586Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:10.2353248Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-11-23T01:13:10.2375315Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:10.2414728Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:10.2434017Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:10.2473279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:10.2490659Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:10.2529770Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-11-23T01:13:10.2552395Z Entering 'third_party/pocketfft'
2022-11-23T01:13:10.2591921Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-11-23T01:13:10.2609367Z Entering 'third_party/protobuf'
2022-11-23T01:13:10.2647932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-11-23T01:13:10.2669524Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:10.2706763Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-11-23T01:13:10.2724808Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:10.2765602Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:10.2784699Z Entering 'third_party/psimd'
2022-11-23T01:13:10.2823960Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-11-23T01:13:10.2841636Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:10.2881213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-11-23T01:13:10.2897892Z Entering 'third_party/pybind11'
2022-11-23T01:13:10.2936564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:10.2954423Z Entering 'third_party/python-enum'
2022-11-23T01:13:10.2992671Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-11-23T01:13:10.3009385Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:10.3047846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-11-23T01:13:10.3064702Z Entering 'third_party/python-six'
2022-11-23T01:13:10.3103712Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-11-23T01:13:10.3120702Z Entering 'third_party/sleef'
2022-11-23T01:13:10.3159086Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-11-23T01:13:10.3176317Z Entering 'third_party/tbb'
2022-11-23T01:13:10.3215458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-11-23T01:13:10.3235102Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:10.3273658Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-11-23T01:13:10.3290730Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:10.3329596Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-11-23T01:13:10.3346376Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:10.3384838Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-11-23T01:13:10.3401873Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:10.3439569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-11-23T01:13:10.3456657Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:10.3495093Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-11-23T01:13:10.3512064Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:10.3550554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-11-23T01:13:10.3569924Z Entering 'third_party/zstd'
2022-11-23T01:13:10.3608516Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-11-23T01:13:10.4573608Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-11-23T01:13:10.4883064Z Entering 'android/libs/fbjni'
2022-11-23T01:13:10.4925018Z Entering 'third_party/FP16'
2022-11-23T01:13:10.4967850Z Entering 'third_party/FXdiv'
2022-11-23T01:13:10.5010179Z Entering 'third_party/NNPACK'
2022-11-23T01:13:10.5053571Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:10.5095720Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:10.5138343Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:10.5193364Z Entering 'third_party/benchmark'
2022-11-23T01:13:10.5235836Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:10.5278928Z Entering 'third_party/cub'
2022-11-23T01:13:10.5320767Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:10.5368486Z Entering 'third_party/cutlass'
2022-11-23T01:13:10.5417709Z Entering 'third_party/eigen'
2022-11-23T01:13:10.5462646Z Entering 'third_party/fbgemm'
2022-11-23T01:13:10.5504799Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:10.5546499Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:10.5589192Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:10.5631107Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:10.5673744Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:10.5718499Z Entering 'third_party/fmt'
2022-11-23T01:13:10.5760488Z Entering 'third_party/foxi'
2022-11-23T01:13:10.5802307Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:10.5844958Z Entering 'third_party/gloo'
2022-11-23T01:13:10.5886986Z Entering 'third_party/googletest'
2022-11-23T01:13:10.5929141Z Entering 'third_party/ideep'
2022-11-23T01:13:10.5970956Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:10.6014809Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:10.6064382Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:10.6105881Z Entering 'third_party/ittapi'
2022-11-23T01:13:10.6148754Z Entering 'third_party/kineto'
2022-11-23T01:13:10.6191518Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:10.6233728Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:10.6276921Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:10.6318996Z Entering 'third_party/neon2sse'
2022-11-23T01:13:10.6360610Z Entering 'third_party/nlohmann'
2022-11-23T01:13:10.6403566Z Entering 'third_party/onnx'
2022-11-23T01:13:10.6457861Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:10.6500641Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:10.6544832Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:10.6585837Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:10.6633637Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:10.6676819Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:10.6717860Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:10.6764126Z Entering 'third_party/pocketfft'
2022-11-23T01:13:10.6806022Z Entering 'third_party/protobuf'
2022-11-23T01:13:10.6851755Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:10.6893369Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:10.6936662Z Entering 'third_party/psimd'
2022-11-23T01:13:10.6979457Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:10.7021100Z Entering 'third_party/pybind11'
2022-11-23T01:13:10.7063170Z Entering 'third_party/python-enum'
2022-11-23T01:13:10.7106060Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:10.7148732Z Entering 'third_party/python-six'
2022-11-23T01:13:10.7191746Z Entering 'third_party/sleef'
2022-11-23T01:13:10.7234408Z Entering 'third_party/tbb'
2022-11-23T01:13:10.7279557Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:10.7321632Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:10.7363978Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:10.7405750Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:10.7448206Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:10.7489285Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:10.7534648Z Entering 'third_party/zstd'
2022-11-23T01:13:10.7590532Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-11-23T01:13:10.7899457Z Entering 'android/libs/fbjni'
2022-11-23T01:13:10.7941109Z Entering 'third_party/FP16'
2022-11-23T01:13:10.7984443Z Entering 'third_party/FXdiv'
2022-11-23T01:13:10.8026619Z Entering 'third_party/NNPACK'
2022-11-23T01:13:10.8069015Z Entering 'third_party/QNNPACK'
2022-11-23T01:13:10.8111058Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T01:13:10.8153618Z Entering 'third_party/XNNPACK'
2022-11-23T01:13:10.8207492Z Entering 'third_party/benchmark'
2022-11-23T01:13:10.8249892Z Entering 'third_party/cpuinfo'
2022-11-23T01:13:10.8292582Z Entering 'third_party/cub'
2022-11-23T01:13:10.8334742Z Entering 'third_party/cudnn_frontend'
2022-11-23T01:13:10.8382446Z Entering 'third_party/cutlass'
2022-11-23T01:13:10.8431689Z Entering 'third_party/eigen'
2022-11-23T01:13:10.8475827Z Entering 'third_party/fbgemm'
2022-11-23T01:13:10.8518021Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T01:13:10.8560409Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T01:13:10.8603547Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T01:13:10.8645666Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T01:13:10.8688558Z Entering 'third_party/flatbuffers'
2022-11-23T01:13:10.8732086Z Entering 'third_party/fmt'
2022-11-23T01:13:10.8774844Z Entering 'third_party/foxi'
2022-11-23T01:13:10.8815983Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T01:13:10.8857988Z Entering 'third_party/gloo'
2022-11-23T01:13:10.8900553Z Entering 'third_party/googletest'
2022-11-23T01:13:10.8943063Z Entering 'third_party/ideep'
2022-11-23T01:13:10.8983979Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T01:13:10.9028022Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T01:13:10.9076934Z Entering 'third_party/ios-cmake'
2022-11-23T01:13:10.9118610Z Entering 'third_party/ittapi'
2022-11-23T01:13:10.9161491Z Entering 'third_party/kineto'
2022-11-23T01:13:10.9203405Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T01:13:10.9245822Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T01:13:10.9289174Z Entering 'third_party/nccl/nccl'
2022-11-23T01:13:10.9331058Z Entering 'third_party/neon2sse'
2022-11-23T01:13:10.9373050Z Entering 'third_party/nlohmann'
2022-11-23T01:13:10.9416396Z Entering 'third_party/onnx'
2022-11-23T01:13:10.9473012Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T01:13:10.9515907Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T01:13:10.9560254Z Entering 'third_party/onnx-tensorrt'
2022-11-23T01:13:10.9601344Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T01:13:10.9648666Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T01:13:10.9690630Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T01:13:10.9733291Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T01:13:10.9780457Z Entering 'third_party/pocketfft'
2022-11-23T01:13:10.9822600Z Entering 'third_party/protobuf'
2022-11-23T01:13:10.9869147Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T01:13:10.9910883Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T01:13:10.9954403Z Entering 'third_party/psimd'
2022-11-23T01:13:10.9995849Z Entering 'third_party/pthreadpool'
2022-11-23T01:13:11.0038794Z Entering 'third_party/pybind11'
2022-11-23T01:13:11.0080628Z Entering 'third_party/python-enum'
2022-11-23T01:13:11.0121509Z Entering 'third_party/python-peachpy'
2022-11-23T01:13:11.0163517Z Entering 'third_party/python-six'
2022-11-23T01:13:11.0204601Z Entering 'third_party/sleef'
2022-11-23T01:13:11.0246727Z Entering 'third_party/tbb'
2022-11-23T01:13:11.0291411Z Entering 'third_party/tensorpipe'
2022-11-23T01:13:11.0333828Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T01:13:11.0376106Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T01:13:11.0418467Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T01:13:11.0459959Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T01:13:11.0500908Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T01:13:11.0545015Z Entering 'third_party/zstd'
2022-11-23T01:13:11.0596180Z ##[endgroup]
2022-11-23T01:13:11.0646433Z [command]/usr/bin/git log -1 --format='%H'
2022-11-23T01:13:11.0674045Z '1cfd3858ac54fe3883534309081631a0a892ba3f'
2022-11-23T01:13:11.0823189Z Prepare all required actions
2022-11-23T01:13:11.0854686Z ##[group]Run ./.github/actions/setup-linux
2022-11-23T01:13:11.0854969Z env:
2022-11-23T01:13:11.0855208Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:11.0855450Z ##[endgroup]
2022-11-23T01:13:11.0873538Z ##[group]Run set -euo pipefail
2022-11-23T01:13:11.0873851Z set -euo pipefail
2022-11-23T01:13:11.0874139Z function get_ec2_metadata() {
2022-11-23T01:13:11.0874477Z   # Pulled from instance metadata endpoint for EC2
2022-11-23T01:13:11.0874935Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-11-23T01:13:11.0875339Z   category=$1
2022-11-23T01:13:11.0875669Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-11-23T01:13:11.0875955Z }
2022-11-23T01:13:11.0876229Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-11-23T01:13:11.0876616Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-11-23T01:13:11.0876979Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-11-23T01:13:11.0877320Z echo "system info $(uname -a)"
2022-11-23T01:13:11.0889904Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:11.0890180Z env:
2022-11-23T01:13:11.0890417Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:11.0890673Z ##[endgroup]
2022-11-23T01:13:11.0991641Z ami-id: ami-096198a0bccc6bad4
2022-11-23T01:13:11.1054322Z instance-id: i-099ed1720beaf2723
2022-11-23T01:13:11.1118782Z instance-type: g3.8xlarge
2022-11-23T01:13:11.1127046Z system info Linux ip-10-0-2-220.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-11-23T01:13:11.1146289Z ##[group]Run if systemctl is-active --quiet docker; then
2022-11-23T01:13:11.1146678Z if systemctl is-active --quiet docker; then
2022-11-23T01:13:11.1147400Z     echo "Docker daemon is running...";
2022-11-23T01:13:11.1147674Z else
2022-11-23T01:13:11.1148002Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-11-23T01:13:11.1148320Z fi
2022-11-23T01:13:11.1159605Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:11.1159907Z env:
2022-11-23T01:13:11.1160151Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:11.1160390Z ##[endgroup]
2022-11-23T01:13:11.1210416Z Docker daemon is running...
2022-11-23T01:13:11.1229015Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-11-23T01:13:11.1229514Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-11-23T01:13:11.1229922Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:13:11.1230430Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-11-23T01:13:11.1230903Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-11-23T01:13:11.1241836Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:11.1242149Z env:
2022-11-23T01:13:11.1242378Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:11.1242656Z   AWS_RETRY_MODE: standard
2022-11-23T01:13:11.1242925Z   AWS_MAX_ATTEMPTS: 5
2022-11-23T01:13:11.1243188Z   AWS_DEFAULT_REGION: us-east-1
2022-11-23T01:13:11.1243457Z ##[endgroup]
2022-11-23T01:13:12.0504618Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-11-23T01:13:12.0505150Z Configure a credential helper to remove this warning. See
2022-11-23T01:13:12.0505693Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-11-23T01:13:12.0505979Z 
2022-11-23T01:13:12.0506614Z Login Succeeded
2022-11-23T01:13:12.0600123Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:13:12.0600538Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:13:12.0601005Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-11-23T01:13:12.0613441Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:12.0613740Z env:
2022-11-23T01:13:12.0613980Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:12.0614218Z ##[endgroup]
2022-11-23T01:13:12.0703182Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2022-11-23T01:13:12.0703533Z with:
2022-11-23T01:13:12.0704018Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:13:12.0704479Z env:
2022-11-23T01:13:12.0704717Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:12.0704954Z ##[endgroup]
2022-11-23T01:13:12.0721393Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:13:12.0721765Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-11-23T01:13:12.0722132Z # ignore output since only exit code is used for conditional
2022-11-23T01:13:12.0722520Z # only pull docker image if it's not available locally
2022-11-23T01:13:12.0722918Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-11-23T01:13:12.0723320Z   retry docker pull "${DOCKER_IMAGE}"
2022-11-23T01:13:12.0723597Z fi
2022-11-23T01:13:12.0735380Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:13:12.0735663Z env:
2022-11-23T01:13:12.0735907Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:13:12.0736429Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:13:12.0736909Z ##[endgroup]
2022-11-23T01:13:12.3415964Z 072aae4a77ed7d3a69ad5683420509c41301b940: Pulling from pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7
2022-11-23T01:13:12.3416717Z a404e5416296: Pulling fs layer
2022-11-23T01:13:12.3417002Z c58c079e9b17: Pulling fs layer
2022-11-23T01:13:12.3417261Z e5b80b8bbe91: Pulling fs layer
2022-11-23T01:13:12.3417543Z 888240790290: Pulling fs layer
2022-11-23T01:13:12.3417818Z 515fe5e34eb4: Pulling fs layer
2022-11-23T01:13:12.3419583Z 4e4521f12f5a: Pulling fs layer
2022-11-23T01:13:12.3419983Z f6e1a56cb32d: Pulling fs layer
2022-11-23T01:13:12.3420277Z c29b96e36bd0: Pulling fs layer
2022-11-23T01:13:12.3420548Z 304d3c6c28d0: Pulling fs layer
2022-11-23T01:13:12.3420817Z fac00e927cfe: Pulling fs layer
2022-11-23T01:13:12.3421090Z f0158c8d8420: Pulling fs layer
2022-11-23T01:13:12.3421362Z 3ceac802dd07: Pulling fs layer
2022-11-23T01:13:12.3421620Z 0d0e625ba887: Pulling fs layer
2022-11-23T01:13:12.3422245Z 888240790290: Waiting
2022-11-23T01:13:12.3422511Z bc2be817cb7e: Pulling fs layer
2022-11-23T01:13:12.3422759Z 515fe5e34eb4: Waiting
2022-11-23T01:13:12.3422999Z 4e4521f12f5a: Waiting
2022-11-23T01:13:12.3423248Z 11eb2106b948: Pulling fs layer
2022-11-23T01:13:12.3423495Z 34fa4193c7a6: Pulling fs layer
2022-11-23T01:13:12.3429078Z c29b96e36bd0: Waiting
2022-11-23T01:13:12.3429460Z a7cf5b3894f8: Pulling fs layer
2022-11-23T01:13:12.3429762Z f6e1a56cb32d: Waiting
2022-11-23T01:13:12.3430028Z 3f6b06edd3f5: Pulling fs layer
2022-11-23T01:13:12.3430273Z f0158c8d8420: Waiting
2022-11-23T01:13:12.3430539Z 73a2b1f75a3d: Pulling fs layer
2022-11-23T01:13:12.3430813Z ba6235196410: Pulling fs layer
2022-11-23T01:13:12.3431061Z 879cdaf83543: Pulling fs layer
2022-11-23T01:13:12.3431318Z 3ceac802dd07: Waiting
2022-11-23T01:13:12.3431567Z 34fa4193c7a6: Waiting
2022-11-23T01:13:12.3431796Z fac00e927cfe: Waiting
2022-11-23T01:13:12.3432039Z a7cf5b3894f8: Waiting
2022-11-23T01:13:12.3432276Z 0d0e625ba887: Waiting
2022-11-23T01:13:12.3432512Z 6ff0fc00b0a9: Pulling fs layer
2022-11-23T01:13:12.3432774Z bc2be817cb7e: Waiting
2022-11-23T01:13:12.3433030Z a58b9ed071f4: Pulling fs layer
2022-11-23T01:13:12.3433342Z a8c562f6a1cf: Pulling fs layer
2022-11-23T01:13:12.3433586Z 11eb2106b948: Waiting
2022-11-23T01:13:12.3433836Z 0a39b4492650: Pulling fs layer
2022-11-23T01:13:12.3434103Z 9088ff8de269: Pulling fs layer
2022-11-23T01:13:12.3434339Z 879cdaf83543: Waiting
2022-11-23T01:13:12.3434772Z 165006759af3: Pulling fs layer
2022-11-23T01:13:12.3435037Z ae48b7377a0d: Pulling fs layer
2022-11-23T01:13:12.3435292Z b18965f4b6f1: Pulling fs layer
2022-11-23T01:13:12.3435625Z 9088ff8de269: Waiting
2022-11-23T01:13:12.3435893Z 102ddcd90753: Pulling fs layer
2022-11-23T01:13:12.3436157Z 5f5dd1cba120: Pulling fs layer
2022-11-23T01:13:12.3436423Z a58b9ed071f4: Waiting
2022-11-23T01:13:12.3436662Z 8a7f50c8b503: Pulling fs layer
2022-11-23T01:13:12.3436931Z 863c35620b44: Pulling fs layer
2022-11-23T01:13:12.3437199Z 183e4209dc37: Pulling fs layer
2022-11-23T01:13:12.3437433Z 102ddcd90753: Waiting
2022-11-23T01:13:12.3437690Z a47cba6c334e: Pulling fs layer
2022-11-23T01:13:12.3437947Z 5f5dd1cba120: Waiting
2022-11-23T01:13:12.3438173Z ba6235196410: Waiting
2022-11-23T01:13:12.3438424Z a9f3d4742233: Pulling fs layer
2022-11-23T01:13:12.3438673Z 183e4209dc37: Waiting
2022-11-23T01:13:12.3438895Z 8a7f50c8b503: Waiting
2022-11-23T01:13:12.3439150Z 3cefa8a4607f: Pulling fs layer
2022-11-23T01:13:12.3439429Z 023a41fa48e6: Pulling fs layer
2022-11-23T01:13:12.3439669Z a47cba6c334e: Waiting
2022-11-23T01:13:12.3439925Z 96e251412f4d: Pulling fs layer
2022-11-23T01:13:12.3440181Z b18965f4b6f1: Waiting
2022-11-23T01:13:12.3440421Z 49d40c00cf56: Pulling fs layer
2022-11-23T01:13:12.3440674Z 023a41fa48e6: Waiting
2022-11-23T01:13:12.3440929Z 7e2d6313145f: Pulling fs layer
2022-11-23T01:13:12.3441163Z 96e251412f4d: Waiting
2022-11-23T01:13:12.3441412Z 96805775a692: Pulling fs layer
2022-11-23T01:13:12.3441674Z 75f1ead35ace: Pulling fs layer
2022-11-23T01:13:12.3441929Z 793c37004dab: Pulling fs layer
2022-11-23T01:13:12.3442175Z 7e2d6313145f: Waiting
2022-11-23T01:13:12.3442432Z cadc5661750d: Pulling fs layer
2022-11-23T01:13:12.3442673Z 3f6b06edd3f5: Waiting
2022-11-23T01:13:12.3443108Z 6386b2adbe28: Pulling fs layer
2022-11-23T01:13:12.3443376Z 74aa250bc82f: Pulling fs layer
2022-11-23T01:13:12.3443608Z 0a39b4492650: Waiting
2022-11-23T01:13:12.3443860Z 436525efe61d: Pulling fs layer
2022-11-23T01:13:12.3444117Z cadc5661750d: Waiting
2022-11-23T01:13:12.3444358Z 596be1fe0bda: Pulling fs layer
2022-11-23T01:13:12.3444626Z 772fa4efddc3: Pulling fs layer
2022-11-23T01:13:12.3444900Z 91ddf385377b: Pulling fs layer
2022-11-23T01:13:12.3445158Z 9f7cfb895784: Pulling fs layer
2022-11-23T01:13:12.3445420Z 8b8218af0479: Pulling fs layer
2022-11-23T01:13:12.3445669Z 6386b2adbe28: Waiting
2022-11-23T01:13:12.3445890Z 74aa250bc82f: Waiting
2022-11-23T01:13:12.3446125Z 165006759af3: Waiting
2022-11-23T01:13:12.3446360Z 9f7cfb895784: Waiting
2022-11-23T01:13:12.3446582Z 772fa4efddc3: Waiting
2022-11-23T01:13:12.3446822Z ae48b7377a0d: Waiting
2022-11-23T01:13:12.3447064Z 596be1fe0bda: Waiting
2022-11-23T01:13:12.3447301Z a8c562f6a1cf: Waiting
2022-11-23T01:13:12.3447515Z 96805775a692: Waiting
2022-11-23T01:13:12.3447750Z 73a2b1f75a3d: Waiting
2022-11-23T01:13:12.3447986Z 436525efe61d: Waiting
2022-11-23T01:13:12.4699984Z c58c079e9b17: Verifying Checksum
2022-11-23T01:13:12.4700322Z c58c079e9b17: Download complete
2022-11-23T01:13:12.5507082Z 888240790290: Verifying Checksum
2022-11-23T01:13:12.5507390Z 888240790290: Download complete
2022-11-23T01:13:12.6327578Z e5b80b8bbe91: Verifying Checksum
2022-11-23T01:13:12.6327904Z e5b80b8bbe91: Download complete
2022-11-23T01:13:12.6330427Z 515fe5e34eb4: Verifying Checksum
2022-11-23T01:13:12.6331038Z 515fe5e34eb4: Download complete
2022-11-23T01:13:12.7152599Z f6e1a56cb32d: Download complete
2022-11-23T01:13:12.7623381Z a404e5416296: Download complete
2022-11-23T01:13:12.8797536Z 304d3c6c28d0: Download complete
2022-11-23T01:13:12.9699270Z fac00e927cfe: Verifying Checksum
2022-11-23T01:13:12.9699599Z fac00e927cfe: Download complete
2022-11-23T01:13:13.5541384Z a404e5416296: Pull complete
2022-11-23T01:13:13.8358344Z c58c079e9b17: Pull complete
2022-11-23T01:13:14.3434702Z e5b80b8bbe91: Pull complete
2022-11-23T01:13:14.4774457Z 888240790290: Pull complete
2022-11-23T01:13:14.6021210Z 515fe5e34eb4: Pull complete
2022-11-23T01:13:15.0213842Z f0158c8d8420: Verifying Checksum
2022-11-23T01:13:15.0214690Z f0158c8d8420: Download complete
2022-11-23T01:13:15.0960266Z 3ceac802dd07: Verifying Checksum
2022-11-23T01:13:15.0960597Z 3ceac802dd07: Download complete
2022-11-23T01:13:15.1722230Z 0d0e625ba887: Verifying Checksum
2022-11-23T01:13:15.1722545Z 0d0e625ba887: Download complete
2022-11-23T01:13:15.2492078Z bc2be817cb7e: Verifying Checksum
2022-11-23T01:13:15.2492633Z bc2be817cb7e: Download complete
2022-11-23T01:13:15.9769862Z 11eb2106b948: Verifying Checksum
2022-11-23T01:13:15.9770749Z 11eb2106b948: Download complete
2022-11-23T01:13:16.0491706Z 34fa4193c7a6: Verifying Checksum
2022-11-23T01:13:16.0491998Z 34fa4193c7a6: Download complete
2022-11-23T01:13:16.1256712Z a7cf5b3894f8: Verifying Checksum
2022-11-23T01:13:16.1257162Z a7cf5b3894f8: Download complete
2022-11-23T01:13:23.8451003Z 4e4521f12f5a: Verifying Checksum
2022-11-23T01:13:23.8451407Z 4e4521f12f5a: Download complete
2022-11-23T01:13:23.9342733Z 73a2b1f75a3d: Verifying Checksum
2022-11-23T01:13:23.9343346Z 73a2b1f75a3d: Download complete
2022-11-23T01:13:24.0014831Z ba6235196410: Verifying Checksum
2022-11-23T01:13:24.0015404Z ba6235196410: Download complete
2022-11-23T01:13:24.0759402Z 879cdaf83543: Verifying Checksum
2022-11-23T01:13:24.0759742Z 879cdaf83543: Download complete
2022-11-23T01:13:24.1458768Z 6ff0fc00b0a9: Verifying Checksum
2022-11-23T01:13:24.1459133Z 6ff0fc00b0a9: Download complete
2022-11-23T01:13:24.2356807Z a58b9ed071f4: Verifying Checksum
2022-11-23T01:13:24.2357105Z a58b9ed071f4: Download complete
2022-11-23T01:13:24.3021016Z a8c562f6a1cf: Verifying Checksum
2022-11-23T01:13:24.3021359Z a8c562f6a1cf: Download complete
2022-11-23T01:13:25.2431661Z 0a39b4492650: Verifying Checksum
2022-11-23T01:13:25.2432270Z 0a39b4492650: Download complete
2022-11-23T01:13:25.3128436Z 9088ff8de269: Download complete
2022-11-23T01:13:25.4458586Z 165006759af3: Verifying Checksum
2022-11-23T01:13:25.4458914Z 165006759af3: Download complete
2022-11-23T01:13:25.5361426Z ae48b7377a0d: Verifying Checksum
2022-11-23T01:13:25.5361964Z ae48b7377a0d: Download complete
2022-11-23T01:13:25.6109363Z b18965f4b6f1: Download complete
2022-11-23T01:13:25.6842611Z 102ddcd90753: Verifying Checksum
2022-11-23T01:13:25.6842968Z 102ddcd90753: Download complete
2022-11-23T01:13:27.1013974Z c29b96e36bd0: Verifying Checksum
2022-11-23T01:13:27.1014327Z c29b96e36bd0: Download complete
2022-11-23T01:13:27.1820537Z 8a7f50c8b503: Verifying Checksum
2022-11-23T01:13:27.1820866Z 8a7f50c8b503: Download complete
2022-11-23T01:13:27.2467804Z 863c35620b44: Verifying Checksum
2022-11-23T01:13:27.6446985Z 863c35620b44: Download complete
2022-11-23T01:13:27.6447310Z 183e4209dc37: Verifying Checksum
2022-11-23T01:13:27.6447579Z 183e4209dc37: Download complete
2022-11-23T01:13:27.6776486Z 5f5dd1cba120: Verifying Checksum
2022-11-23T01:13:27.6776897Z 5f5dd1cba120: Download complete
2022-11-23T01:13:27.7423824Z a47cba6c334e: Verifying Checksum
2022-11-23T01:13:27.7424396Z a47cba6c334e: Download complete
2022-11-23T01:13:27.7557270Z a9f3d4742233: Download complete
2022-11-23T01:13:27.8316915Z 023a41fa48e6: Verifying Checksum
2022-11-23T01:13:27.8317240Z 023a41fa48e6: Download complete
2022-11-23T01:13:27.9921720Z 3cefa8a4607f: Verifying Checksum
2022-11-23T01:13:27.9922386Z 3cefa8a4607f: Download complete
2022-11-23T01:13:28.0626756Z 49d40c00cf56: Verifying Checksum
2022-11-23T01:13:28.0627534Z 49d40c00cf56: Download complete
2022-11-23T01:13:28.1306753Z 7e2d6313145f: Verifying Checksum
2022-11-23T01:13:28.1307319Z 7e2d6313145f: Download complete
2022-11-23T01:13:28.2866259Z 96e251412f4d: Verifying Checksum
2022-11-23T01:13:28.2866602Z 96e251412f4d: Download complete
2022-11-23T01:13:28.3721818Z 75f1ead35ace: Verifying Checksum
2022-11-23T01:13:28.3722150Z 75f1ead35ace: Download complete
2022-11-23T01:13:28.4810131Z 793c37004dab: Download complete
2022-11-23T01:13:28.5767634Z cadc5661750d: Download complete
2022-11-23T01:13:28.6444713Z 6386b2adbe28: Download complete
2022-11-23T01:13:28.8312491Z 74aa250bc82f: Verifying Checksum
2022-11-23T01:13:28.8313083Z 74aa250bc82f: Download complete
2022-11-23T01:13:28.8950161Z 436525efe61d: Verifying Checksum
2022-11-23T01:13:28.8950503Z 436525efe61d: Download complete
2022-11-23T01:13:29.4987213Z 596be1fe0bda: Verifying Checksum
2022-11-23T01:13:29.4987610Z 596be1fe0bda: Download complete
2022-11-23T01:13:29.5740369Z 772fa4efddc3: Verifying Checksum
2022-11-23T01:13:29.5740695Z 772fa4efddc3: Download complete
2022-11-23T01:13:32.7844919Z 96805775a692: Download complete
2022-11-23T01:13:32.8672217Z 9f7cfb895784: Verifying Checksum
2022-11-23T01:13:32.8672581Z 9f7cfb895784: Download complete
2022-11-23T01:13:32.9290952Z 8b8218af0479: Verifying Checksum
2022-11-23T01:13:32.9291250Z 8b8218af0479: Download complete
2022-11-23T01:13:35.9051729Z 3f6b06edd3f5: Verifying Checksum
2022-11-23T01:13:35.9052078Z 3f6b06edd3f5: Download complete
2022-11-23T01:13:37.8827076Z 4e4521f12f5a: Pull complete
2022-11-23T01:13:37.9830831Z f6e1a56cb32d: Pull complete
2022-11-23T01:13:59.4859177Z c29b96e36bd0: Pull complete
2022-11-23T01:13:59.5681427Z 91ddf385377b: Verifying Checksum
2022-11-23T01:13:59.5683962Z 91ddf385377b: Download complete
2022-11-23T01:14:01.3338374Z 304d3c6c28d0: Pull complete
2022-11-23T01:14:03.1763544Z fac00e927cfe: Pull complete
2022-11-23T01:14:11.0782835Z f0158c8d8420: Pull complete
2022-11-23T01:14:12.9241073Z 3ceac802dd07: Pull complete
2022-11-23T01:14:14.7671257Z 0d0e625ba887: Pull complete
2022-11-23T01:14:16.6211802Z bc2be817cb7e: Pull complete
2022-11-23T01:14:20.9013689Z 11eb2106b948: Pull complete
2022-11-23T01:14:22.7104536Z 34fa4193c7a6: Pull complete
2022-11-23T01:14:24.5585978Z a7cf5b3894f8: Pull complete
2022-11-23T01:14:59.8882644Z 3f6b06edd3f5: Pull complete
2022-11-23T01:14:59.9810925Z 73a2b1f75a3d: Pull complete
2022-11-23T01:15:00.0664314Z ba6235196410: Pull complete
2022-11-23T01:15:00.1921610Z 879cdaf83543: Pull complete
2022-11-23T01:15:00.3132578Z 6ff0fc00b0a9: Pull complete
2022-11-23T01:15:00.4291680Z a58b9ed071f4: Pull complete
2022-11-23T01:15:00.5325679Z a8c562f6a1cf: Pull complete
2022-11-23T01:15:02.8947141Z 0a39b4492650: Pull complete
2022-11-23T01:15:03.0183672Z 9088ff8de269: Pull complete
2022-11-23T01:15:03.1459759Z 165006759af3: Pull complete
2022-11-23T01:15:03.2973432Z ae48b7377a0d: Pull complete
2022-11-23T01:15:03.4105182Z b18965f4b6f1: Pull complete
2022-11-23T01:15:03.5283300Z 102ddcd90753: Pull complete
2022-11-23T01:15:11.2249940Z 5f5dd1cba120: Pull complete
2022-11-23T01:15:13.1166628Z 8a7f50c8b503: Pull complete
2022-11-23T01:15:14.9582297Z 863c35620b44: Pull complete
2022-11-23T01:15:17.7807030Z 183e4209dc37: Pull complete
2022-11-23T01:15:19.6116029Z a47cba6c334e: Pull complete
2022-11-23T01:15:21.8966528Z a9f3d4742233: Pull complete
2022-11-23T01:15:25.0436505Z 3cefa8a4607f: Pull complete
2022-11-23T01:15:27.4559551Z 023a41fa48e6: Pull complete
2022-11-23T01:15:32.6088014Z 96e251412f4d: Pull complete
2022-11-23T01:15:34.9497198Z 49d40c00cf56: Pull complete
2022-11-23T01:15:37.4666308Z 7e2d6313145f: Pull complete
2022-11-23T01:15:45.9949025Z 96805775a692: Pull complete
2022-11-23T01:15:48.5679971Z 75f1ead35ace: Pull complete
2022-11-23T01:15:50.4154025Z 793c37004dab: Pull complete
2022-11-23T01:15:52.2607054Z cadc5661750d: Pull complete
2022-11-23T01:15:54.3366222Z 6386b2adbe28: Pull complete
2022-11-23T01:15:57.4846722Z 74aa250bc82f: Pull complete
2022-11-23T01:15:59.3624754Z 436525efe61d: Pull complete
2022-11-23T01:16:03.2202225Z 596be1fe0bda: Pull complete
2022-11-23T01:16:03.4374233Z 772fa4efddc3: Pull complete
2022-11-23T01:16:46.1341989Z 91ddf385377b: Pull complete
2022-11-23T01:16:47.9827206Z 9f7cfb895784: Pull complete
2022-11-23T01:16:49.8282112Z 8b8218af0479: Pull complete
2022-11-23T01:16:51.1149760Z Digest: sha256:3a5626edfb2c43fb24303351be75287af92426b6bb7c6df2defc98f980346c6a
2022-11-23T01:16:51.6146098Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:16:51.8972531Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:16:51.9074883Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2022-11-23T01:16:51.9075235Z with:
2022-11-23T01:16:51.9075482Z   driver-version: 515.76
2022-11-23T01:16:51.9075703Z env:
2022-11-23T01:16:51.9075944Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:16:51.9076208Z ##[endgroup]
2022-11-23T01:16:52.0742453Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-11-23T01:16:52.0742750Z with:
2022-11-23T01:16:52.0742983Z   timeout_minutes: 10
2022-11-23T01:16:52.0743229Z   max_attempts: 3
2022-11-23T01:16:52.0749670Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"
YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x

        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms

        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            sudo yum groupinstall -y "Development Tools"
            # ensure our kernel install is the same as our underlying kernel,
            # groupinstall "Development Tools" has a habit of mismatching kernel headers
            sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
            sudo modprobe backlight
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi

        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e
            nvidia-smi
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-11-23T01:16:52.0755830Z   retry_wait_seconds: 10
2022-11-23T01:16:52.0756090Z   polling_interval_seconds: 1
2022-11-23T01:16:52.0756369Z   warning_on_retry: true
2022-11-23T01:16:52.0756636Z   continue_on_error: false
2022-11-23T01:16:52.0756860Z env:
2022-11-23T01:16:52.0757098Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:16:52.0757369Z   DRIVER_VERSION: 515.76
2022-11-23T01:16:52.0757601Z ##[endgroup]
2022-11-23T01:16:52.1329439Z 
2022-11-23T01:16:52.1352506Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:16:52.1395603Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.76.run ==
2022-11-23T01:16:52.1397780Z + sudo yum remove -y nvidia-driver-latest-dkms
2022-11-23T01:16:52.6071480Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:16:52.6497486Z No Match for argument: nvidia-driver-latest-dkms
2022-11-23T01:16:52.6784081Z No Packages marked for removal
2022-11-23T01:16:52.6941101Z + echo 'Before installing NVIDIA driver'
2022-11-23T01:16:52.6941415Z + lspci
2022-11-23T01:16:52.6942029Z Before installing NVIDIA driver
2022-11-23T01:16:53.9725916Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2022-11-23T01:16:53.9726544Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2022-11-23T01:16:53.9726970Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2022-11-23T01:16:53.9727350Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2022-11-23T01:16:53.9727727Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2022-11-23T01:16:53.9730060Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2022-11-23T01:16:53.9731621Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:16:53.9732225Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:16:53.9732658Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2022-11-23T01:16:53.9732958Z + lsmod
2022-11-23T01:16:53.9748590Z Module                  Size  Used by
2022-11-23T01:16:53.9749144Z xt_conntrack           16384  1
2022-11-23T01:16:53.9749725Z ipt_MASQUERADE         16384  1
2022-11-23T01:16:53.9750337Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2022-11-23T01:16:53.9750660Z nf_conntrack_netlink    49152  0
2022-11-23T01:16:53.9750972Z nfnetlink              16384  2 nf_conntrack_netlink
2022-11-23T01:16:53.9754398Z xfrm_user              45056  1
2022-11-23T01:16:53.9754915Z xfrm_algo              16384  1 xfrm_user
2022-11-23T01:16:53.9755530Z xt_addrtype            16384  2
2022-11-23T01:16:53.9756077Z iptable_filter         16384  1
2022-11-23T01:16:53.9756448Z iptable_nat            16384  1
2022-11-23T01:16:53.9756704Z nf_conntrack_ipv4      16384  3
2022-11-23T01:16:53.9757000Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2022-11-23T01:16:53.9757302Z nf_nat_ipv4            16384  1 iptable_nat
2022-11-23T01:16:53.9757608Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2022-11-23T01:16:53.9758087Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2022-11-23T01:16:53.9758628Z br_netfilter           24576  0
2022-11-23T01:16:53.9759107Z bridge                172032  1 br_netfilter
2022-11-23T01:16:53.9759405Z stp                    16384  1 bridge
2022-11-23T01:16:53.9759670Z llc                    16384  2 bridge,stp
2022-11-23T01:16:53.9759937Z overlay                86016  0
2022-11-23T01:16:53.9760175Z sunrpc                393216  1
2022-11-23T01:16:53.9760420Z dm_mirror              28672  0
2022-11-23T01:16:53.9760713Z dm_region_hash         20480  1 dm_mirror
2022-11-23T01:16:53.9761010Z dm_log                 20480  2 dm_region_hash,dm_mirror
2022-11-23T01:16:53.9761314Z dm_mod                143360  2 dm_log,dm_mirror
2022-11-23T01:16:53.9761598Z dax                    69632  1 dm_mod
2022-11-23T01:16:53.9761829Z sb_edac                24576  0
2022-11-23T01:16:53.9762095Z crc32_pclmul           16384  0
2022-11-23T01:16:53.9762347Z ghash_clmulni_intel    16384  0
2022-11-23T01:16:53.9762600Z pcbc                   16384  0
2022-11-23T01:16:53.9762856Z aesni_intel           188416  0
2022-11-23T01:16:53.9763119Z ata_piix               36864  0
2022-11-23T01:16:53.9763365Z aes_x86_64             20480  1 aesni_intel
2022-11-23T01:16:53.9763644Z libata                266240  1 ata_piix
2022-11-23T01:16:53.9763916Z crypto_simd            16384  1 aesni_intel
2022-11-23T01:16:53.9764207Z glue_helper            16384  1 aesni_intel
2022-11-23T01:16:53.9764457Z mousedev               24576  0
2022-11-23T01:16:53.9764782Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2022-11-23T01:16:53.9765116Z pcc_cpufreq            16384  0
2022-11-23T01:16:53.9765367Z scsi_mod              245760  1 libata
2022-11-23T01:16:53.9765629Z psmouse                32768  0
2022-11-23T01:16:53.9765879Z evdev                  20480  3
2022-11-23T01:16:53.9766123Z button                 16384  0
2022-11-23T01:16:53.9766364Z ena                   114688  0
2022-11-23T01:16:53.9766615Z xen_blkfront           49152  2
2022-11-23T01:16:53.9766859Z crc32c_intel           24576  0
2022-11-23T01:16:53.9767102Z autofs4                49152  2
2022-11-23T01:16:53.9767350Z + modinfo nvidia
2022-11-23T01:16:53.9767618Z modinfo: ERROR: Module nvidia not found.
2022-11-23T01:16:53.9767881Z + true
2022-11-23T01:16:53.9768112Z + HAS_NVIDIA_DRIVER=0
2022-11-23T01:16:53.9768465Z ++ command -v nvidia-smi
2022-11-23T01:16:53.9768757Z + '[' -x '' ']'
2022-11-23T01:16:53.9769032Z + '[' 0 -eq 0 ']'
2022-11-23T01:16:53.9769359Z + sudo yum groupinstall -y 'Development Tools'
2022-11-23T01:16:54.4481043Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:16:54.7411497Z Resolving Dependencies
2022-11-23T01:16:54.7416403Z --> Running transaction check
2022-11-23T01:16:54.7419474Z ---> Package autoconf.noarch 0:2.69-11.amzn2 will be installed
2022-11-23T01:16:54.7631970Z --> Processing Dependency: m4 >= 1.4.14 for package: autoconf-2.69-11.amzn2.noarch
2022-11-23T01:16:54.9685519Z --> Processing Dependency: perl(Data::Dumper) for package: autoconf-2.69-11.amzn2.noarch
2022-11-23T01:16:54.9687102Z ---> Package automake.noarch 0:1.13.4-3.1.amzn2 will be installed
2022-11-23T01:16:54.9731628Z --> Processing Dependency: perl(Thread::Queue) for package: automake-1.13.4-3.1.amzn2.noarch
2022-11-23T01:16:54.9738366Z --> Processing Dependency: perl(TAP::Parser) for package: automake-1.13.4-3.1.amzn2.noarch
2022-11-23T01:16:54.9749499Z ---> Package bison.x86_64 0:3.0.4-6.amzn2.0.2 will be installed
2022-11-23T01:16:54.9866671Z ---> Package byacc.x86_64 0:1.9.20130304-3.amzn2.0.2 will be installed
2022-11-23T01:16:54.9874016Z ---> Package cscope.x86_64 0:15.8-10.amzn2.0.2 will be installed
2022-11-23T01:16:54.9920196Z --> Processing Dependency: emacs-filesystem for package: cscope-15.8-10.amzn2.0.2.x86_64
2022-11-23T01:16:54.9943646Z ---> Package ctags.x86_64 0:5.8-13.amzn2.0.2 will be installed
2022-11-23T01:16:54.9952783Z ---> Package diffstat.x86_64 0:1.57-4.amzn2.0.2 will be installed
2022-11-23T01:16:54.9960426Z ---> Package doxygen.x86_64 1:1.8.5-4.amzn2 will be installed
2022-11-23T01:16:55.0061341Z ---> Package elfutils.x86_64 0:0.176-2.amzn2 will be installed
2022-11-23T01:16:55.0193726Z ---> Package flex.x86_64 0:2.5.37-3.amzn2.0.3 will be installed
2022-11-23T01:16:55.0212406Z ---> Package gcc.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:55.0381804Z --> Processing Dependency: cpp = 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:55.0401319Z --> Processing Dependency: libsanitizer >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:55.0454709Z --> Processing Dependency: libquadmath >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:55.0505917Z --> Processing Dependency: libmpx >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:55.0560108Z --> Processing Dependency: libitm >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:55.0611532Z --> Processing Dependency: libcilkrts >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:55.0663631Z --> Processing Dependency: libatomic >= 7.3.1-15.amzn2 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:55.0717856Z --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:55.0879567Z --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:55.0900353Z --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:55.0921480Z ---> Package gcc-c++.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:55.0947960Z ---> Package gcc-gfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:55.0981443Z --> Processing Dependency: libgfortran.so.4()(64bit) for package: gcc-gfortran-7.3.1-15.amzn2.x86_64
2022-11-23T01:16:55.1043205Z ---> Package indent.x86_64 0:2.2.11-13.amzn2.0.2 will be installed
2022-11-23T01:16:55.1057387Z ---> Package intltool.noarch 0:0.50.2-7.amzn2 will be installed
2022-11-23T01:16:55.1107486Z --> Processing Dependency: perl(XML::Parser) for package: intltool-0.50.2-7.amzn2.noarch
2022-11-23T01:16:55.1122774Z --> Processing Dependency: gettext-devel for package: intltool-0.50.2-7.amzn2.noarch
2022-11-23T01:16:55.1141096Z ---> Package libtool.x86_64 0:2.4.2-22.2.amzn2.0.2 will be installed
2022-11-23T01:16:55.1169991Z ---> Package patch.x86_64 0:2.7.1-12.amzn2.0.2 will be installed
2022-11-23T01:16:55.1206040Z ---> Package patchutils.x86_64 0:0.3.3-4.amzn2.0.1 will be installed
2022-11-23T01:16:55.1230942Z ---> Package rcs.x86_64 0:5.9.0-5.amzn2.0.2 will be installed
2022-11-23T01:16:55.1262521Z ---> Package rpm-build.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-11-23T01:16:55.1497256Z --> Processing Dependency: /usr/bin/gdb-add-index for package: rpm-build-4.11.3-48.amzn2.0.2.x86_64
2022-11-23T01:16:55.1514752Z ---> Package rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2 will be installed
2022-11-23T01:16:55.1537777Z ---> Package subversion.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-11-23T01:16:55.1705050Z --> Processing Dependency: subversion-libs(x86-64) = 1.7.14-16.amzn2.0.1 for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1724681Z --> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1726000Z --> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1727218Z --> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1728512Z --> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1729762Z --> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1731021Z --> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1732224Z --> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1733442Z --> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1734102Z --> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1734729Z --> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1735342Z --> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1735967Z --> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1736602Z --> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1737230Z --> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1737823Z --> Processing Dependency: libneon.so.27()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1753296Z --> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1773451Z --> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-16.amzn2.0.1.x86_64
2022-11-23T01:16:55.1797537Z ---> Package swig.x86_64 0:3.0.12-11.amzn2.0.3 will be installed
2022-11-23T01:16:55.1818081Z ---> Package system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14 will be installed
2022-11-23T01:16:55.1864287Z --> Processing Dependency: dwz >= 0.4 for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-11-23T01:16:55.1881511Z --> Processing Dependency: perl-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-11-23T01:16:55.1893729Z --> Processing Dependency: go-srpm-macros for package: system-rpm-config-9.1.0-76.amzn2.0.14.noarch
2022-11-23T01:16:55.2068732Z ---> Package systemtap.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-11-23T01:16:55.2081501Z --> Processing Dependency: systemtap-devel = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:16:55.2096480Z --> Processing Dependency: systemtap-client = 4.5-1.amzn2.0.1 for package: systemtap-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:16:55.2110095Z --> Running transaction check
2022-11-23T01:16:55.2113496Z ---> Package apr.x86_64 0:1.7.0-9.amzn2 will be installed
2022-11-23T01:16:55.2192671Z ---> Package apr-util.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-11-23T01:16:55.2230769Z --> Processing Dependency: apr-util-bdb(x86-64) = 1.6.1-5.amzn2.0.2 for package: apr-util-1.6.1-5.amzn2.0.2.x86_64
2022-11-23T01:16:55.2244507Z ---> Package cpp.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:55.2316903Z ---> Package dwz.x86_64 0:0.11-3.amzn2.0.3 will be installed
2022-11-23T01:16:55.2327615Z ---> Package emacs-filesystem.noarch 1:27.2-4.amzn2.0.1 will be installed
2022-11-23T01:16:55.2328826Z ---> Package gdb.x86_64 0:8.0.1-36.amzn2.0.1 will be installed
2022-11-23T01:16:55.2397626Z ---> Package gettext-devel.x86_64 0:0.19.8.1-3.amzn2 will be installed
2022-11-23T01:16:55.2454538Z --> Processing Dependency: gettext-common-devel = 0.19.8.1-3.amzn2 for package: gettext-devel-0.19.8.1-3.amzn2.x86_64
2022-11-23T01:16:55.2463255Z ---> Package glibc-devel.x86_64 0:2.26-62.amzn2 will be installed
2022-11-23T01:16:55.2583175Z --> Processing Dependency: glibc-headers = 2.26-62.amzn2 for package: glibc-devel-2.26-62.amzn2.x86_64
2022-11-23T01:16:55.2611690Z --> Processing Dependency: glibc-headers for package: glibc-devel-2.26-62.amzn2.x86_64
2022-11-23T01:16:55.2612271Z ---> Package go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.2 will be installed
2022-11-23T01:16:55.2616987Z ---> Package libatomic.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:55.2630553Z ---> Package libcilkrts.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:55.2656746Z ---> Package libgfortran.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:55.2691873Z ---> Package libitm.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:55.2707489Z ---> Package libmpc.x86_64 0:1.0.1-3.amzn2.0.2 will be installed
2022-11-23T01:16:55.2719981Z ---> Package libmpx.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:55.2734688Z ---> Package libquadmath.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:55.2759906Z ---> Package libsanitizer.x86_64 0:7.3.1-15.amzn2 will be installed
2022-11-23T01:16:55.2804853Z ---> Package m4.x86_64 0:1.4.16-10.amzn2.0.2 will be installed
2022-11-23T01:16:55.2819505Z ---> Package mpfr.x86_64 0:3.1.1-4.amzn2.0.2 will be installed
2022-11-23T01:16:55.2840538Z ---> Package neon.x86_64 0:0.30.0-3.amzn2.0.2 will be installed
2022-11-23T01:16:55.2914120Z --> Processing Dependency: libgnutls.so.28(GNUTLS_2_12)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:16:55.2951844Z --> Processing Dependency: libgnutls.so.28(GNUTLS_1_4)(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:16:55.2953203Z --> Processing Dependency: libproxy.so.1()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:16:55.2971506Z --> Processing Dependency: libpakchois.so.0()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:16:55.2988493Z --> Processing Dependency: libgnutls.so.28()(64bit) for package: neon-0.30.0-3.amzn2.0.2.x86_64
2022-11-23T01:16:55.2994906Z ---> Package perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2 will be installed
2022-11-23T01:16:55.3041710Z ---> Package perl-Test-Harness.noarch 0:3.28-3.amzn2 will be installed
2022-11-23T01:16:55.3134224Z ---> Package perl-Thread-Queue.noarch 0:3.02-2.amzn2 will be installed
2022-11-23T01:16:55.3145993Z ---> Package perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2 will be installed
2022-11-23T01:16:55.3161964Z ---> Package perl-srpm-macros.noarch 0:1-8.amzn2.0.1 will be installed
2022-11-23T01:16:55.3163266Z ---> Package subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1 will be installed
2022-11-23T01:16:55.3190838Z ---> Package systemtap-client.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-11-23T01:16:55.3394573Z --> Processing Dependency: mokutil for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:16:55.3407691Z --> Processing Dependency: libavahi-common.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:16:55.3433862Z --> Processing Dependency: libavahi-client.so.3()(64bit) for package: systemtap-client-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:16:55.3434422Z ---> Package systemtap-devel.x86_64 0:4.5-1.amzn2.0.1 will be installed
2022-11-23T01:16:55.3547386Z --> Processing Dependency: kernel-devel-uname-r for package: systemtap-devel-4.5-1.amzn2.0.1.x86_64
2022-11-23T01:16:55.4572189Z --> Running transaction check
2022-11-23T01:16:55.4573348Z ---> Package apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2 will be installed
2022-11-23T01:16:55.4582771Z ---> Package avahi-libs.x86_64 0:0.6.31-20.amzn2 will be installed
2022-11-23T01:16:55.4608320Z ---> Package gettext-common-devel.noarch 0:0.19.8.1-3.amzn2 will be installed
2022-11-23T01:16:55.4609558Z ---> Package glibc-headers.x86_64 0:2.26-62.amzn2 will be installed
2022-11-23T01:16:55.4682751Z --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.26-62.amzn2.x86_64
2022-11-23T01:16:55.5782702Z --> Processing Dependency: kernel-headers for package: glibc-headers-2.26-62.amzn2.x86_64
2022-11-23T01:16:55.5783447Z ---> Package gnutls.x86_64 0:3.3.29-9.amzn2.0.1 will be installed
2022-11-23T01:16:55.5848512Z --> Processing Dependency: trousers >= 0.3.11.2 for package: gnutls-3.3.29-9.amzn2.0.1.x86_64
2022-11-23T01:16:55.5874616Z ---> Package kernel-devel.x86_64 0:4.14.296-222.539.amzn2 will be installed
2022-11-23T01:16:55.5900388Z --> Processing Dependency: elfutils-libelf-devel for package: kernel-devel-4.14.296-222.539.amzn2.x86_64
2022-11-23T01:16:55.5919956Z ---> Package libproxy.x86_64 0:0.4.11-10.amzn2.0.3 will be installed
2022-11-23T01:16:55.5947706Z --> Processing Dependency: libmodman.so.1()(64bit) for package: libproxy-0.4.11-10.amzn2.0.3.x86_64
2022-11-23T01:16:55.5965607Z ---> Package mokutil.x86_64 1:0.3.0-10.amzn2.0.1 will be installed
2022-11-23T01:16:55.6013227Z --> Processing Dependency: libefivar.so.1(libefivar.so.0)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-11-23T01:16:55.6033665Z --> Processing Dependency: libefivar.so.1(LIBEFIVAR_0.24)(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-11-23T01:16:55.6034969Z --> Processing Dependency: libefivar.so.1()(64bit) for package: 1:mokutil-0.3.0-10.amzn2.0.1.x86_64
2022-11-23T01:16:55.6035572Z ---> Package pakchois.x86_64 0:0.4-10.amzn2.0.2 will be installed
2022-11-23T01:16:55.6048681Z --> Running transaction check
2022-11-23T01:16:55.6049647Z ---> Package efivar-libs.x86_64 0:31-4.amzn2.0.4 will be installed
2022-11-23T01:16:55.6066568Z ---> Package elfutils-libelf-devel.x86_64 0:0.176-2.amzn2 will be installed
2022-11-23T01:16:55.6078887Z --> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.amzn2.x86_64
2022-11-23T01:16:55.6106920Z ---> Package kernel-headers.x86_64 0:4.14.296-222.539.amzn2 will be installed
2022-11-23T01:16:55.6108427Z ---> Package libmodman.x86_64 0:2.0.1-8.amzn2.0.2 will be installed
2022-11-23T01:16:55.6125853Z ---> Package trousers.x86_64 0:0.3.14-2.amzn2.0.2 will be installed
2022-11-23T01:16:55.6181857Z --> Running transaction check
2022-11-23T01:16:55.6182911Z ---> Package zlib-devel.x86_64 0:1.2.7-19.amzn2.0.2 will be installed
2022-11-23T01:16:55.8758480Z --> Finished Dependency Resolution
2022-11-23T01:16:55.9507463Z 
2022-11-23T01:16:55.9507922Z Dependencies Resolved
2022-11-23T01:16:55.9617997Z 
2022-11-23T01:16:55.9618249Z ================================================================================
2022-11-23T01:16:55.9618616Z  Package                 Arch     Version                    Repository    Size
2022-11-23T01:16:55.9618973Z ================================================================================
2022-11-23T01:16:55.9619289Z Installing for group install "Development Tools":
2022-11-23T01:16:55.9619812Z  autoconf                noarch   2.69-11.amzn2              amzn2-core   701 k
2022-11-23T01:16:55.9620567Z  automake                noarch   1.13.4-3.1.amzn2           amzn2-core   679 k
2022-11-23T01:16:55.9621217Z  bison                   x86_64   3.0.4-6.amzn2.0.2          amzn2-core   674 k
2022-11-23T01:16:55.9621663Z  byacc                   x86_64   1.9.20130304-3.amzn2.0.2   amzn2-core    66 k
2022-11-23T01:16:55.9622107Z  cscope                  x86_64   15.8-10.amzn2.0.2          amzn2-core   204 k
2022-11-23T01:16:55.9622814Z  ctags                   x86_64   5.8-13.amzn2.0.2           amzn2-core   157 k
2022-11-23T01:16:55.9623272Z  diffstat                x86_64   1.57-4.amzn2.0.2           amzn2-core    35 k
2022-11-23T01:16:55.9623691Z  doxygen                 x86_64   1:1.8.5-4.amzn2            amzn2-core   3.5 M
2022-11-23T01:16:55.9624119Z  elfutils                x86_64   0.176-2.amzn2              amzn2-core   307 k
2022-11-23T01:16:55.9624549Z  flex                    x86_64   2.5.37-3.amzn2.0.3         amzn2-core   291 k
2022-11-23T01:16:55.9624958Z  gcc                     x86_64   7.3.1-15.amzn2             amzn2-core    22 M
2022-11-23T01:16:55.9625431Z  gcc-c++                 x86_64   7.3.1-15.amzn2             amzn2-core    13 M
2022-11-23T01:16:55.9625879Z  gcc-gfortran            x86_64   7.3.1-15.amzn2             amzn2-core    11 M
2022-11-23T01:16:55.9626316Z  indent                  x86_64   2.2.11-13.amzn2.0.2        amzn2-core   150 k
2022-11-23T01:16:55.9626749Z  intltool                noarch   0.50.2-7.amzn2             amzn2-core    59 k
2022-11-23T01:16:55.9627740Z  libtool                 x86_64   2.4.2-22.2.amzn2.0.2       amzn2-core   588 k
2022-11-23T01:16:55.9628439Z  patch                   x86_64   2.7.1-12.amzn2.0.2         amzn2-core   110 k
2022-11-23T01:16:55.9628878Z  patchutils              x86_64   0.3.3-4.amzn2.0.1          amzn2-core   104 k
2022-11-23T01:16:55.9629303Z  rcs                     x86_64   5.9.0-5.amzn2.0.2          amzn2-core   231 k
2022-11-23T01:16:55.9629724Z  rpm-build               x86_64   4.11.3-48.amzn2.0.2        amzn2-core   150 k
2022-11-23T01:16:55.9630166Z  rpm-sign                x86_64   4.11.3-48.amzn2.0.2        amzn2-core    50 k
2022-11-23T01:16:55.9630588Z  subversion              x86_64   1.7.14-16.amzn2.0.1        amzn2-core   1.0 M
2022-11-23T01:16:55.9631015Z  swig                    x86_64   3.0.12-11.amzn2.0.3        amzn2-core   1.4 M
2022-11-23T01:16:55.9631475Z  system-rpm-config       noarch   9.1.0-76.amzn2.0.14        amzn2-core    90 k
2022-11-23T01:16:55.9631915Z  systemtap               x86_64   4.5-1.amzn2.0.1            amzn2-core    12 k
2022-11-23T01:16:55.9632231Z Installing for dependencies:
2022-11-23T01:16:55.9632648Z  apr                     x86_64   1.7.0-9.amzn2              amzn2-core   122 k
2022-11-23T01:16:55.9633119Z  apr-util                x86_64   1.6.1-5.amzn2.0.2          amzn2-core    99 k
2022-11-23T01:16:55.9633539Z  apr-util-bdb            x86_64   1.6.1-5.amzn2.0.2          amzn2-core    19 k
2022-11-23T01:16:55.9633982Z  avahi-libs              x86_64   0.6.31-20.amzn2            amzn2-core    61 k
2022-11-23T01:16:55.9634413Z  cpp                     x86_64   7.3.1-15.amzn2             amzn2-core   9.2 M
2022-11-23T01:16:55.9634819Z  dwz                     x86_64   0.11-3.amzn2.0.3           amzn2-core    98 k
2022-11-23T01:16:55.9635253Z  efivar-libs             x86_64   31-4.amzn2.0.4             amzn2-core    68 k
2022-11-23T01:16:55.9635718Z  elfutils-libelf-devel   x86_64   0.176-2.amzn2              amzn2-core    40 k
2022-11-23T01:16:55.9636191Z  emacs-filesystem        noarch   1:27.2-4.amzn2.0.1         amzn2-core    67 k
2022-11-23T01:16:55.9636617Z  gdb                     x86_64   8.0.1-36.amzn2.0.1         amzn2-core   3.1 M
2022-11-23T01:16:55.9637065Z  gettext-common-devel    noarch   0.19.8.1-3.amzn2           amzn2-core   410 k
2022-11-23T01:16:55.9637532Z  gettext-devel           x86_64   0.19.8.1-3.amzn2           amzn2-core   320 k
2022-11-23T01:16:55.9637952Z  glibc-devel             x86_64   2.26-62.amzn2              amzn2-core   995 k
2022-11-23T01:16:55.9638398Z  glibc-headers           x86_64   2.26-62.amzn2              amzn2-core   516 k
2022-11-23T01:16:55.9638832Z  gnutls                  x86_64   3.3.29-9.amzn2.0.1         amzn2-core   661 k
2022-11-23T01:16:55.9639272Z  go-srpm-macros          noarch   3.0.15-23.amzn2.0.2        amzn2-core    23 k
2022-11-23T01:16:55.9639713Z  kernel-devel            x86_64   4.14.296-222.539.amzn2     amzn2-core    13 M
2022-11-23T01:16:55.9640363Z  kernel-headers          x86_64   4.14.296-222.539.amzn2     amzn2-core   1.2 M
2022-11-23T01:16:55.9640823Z  libatomic               x86_64   7.3.1-15.amzn2             amzn2-core    46 k
2022-11-23T01:16:55.9641238Z  libcilkrts              x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-11-23T01:16:55.9641669Z  libgfortran             x86_64   7.3.1-15.amzn2             amzn2-core   536 k
2022-11-23T01:16:55.9642090Z  libitm                  x86_64   7.3.1-15.amzn2             amzn2-core    85 k
2022-11-23T01:16:55.9642523Z  libmodman               x86_64   2.0.1-8.amzn2.0.2          amzn2-core    29 k
2022-11-23T01:16:55.9642938Z  libmpc                  x86_64   1.0.1-3.amzn2.0.2          amzn2-core    52 k
2022-11-23T01:16:55.9643360Z  libmpx                  x86_64   7.3.1-15.amzn2             amzn2-core    51 k
2022-11-23T01:16:55.9643794Z  libproxy                x86_64   0.4.11-10.amzn2.0.3        amzn2-core    61 k
2022-11-23T01:16:55.9644223Z  libquadmath             x86_64   7.3.1-15.amzn2             amzn2-core   189 k
2022-11-23T01:16:55.9644665Z  libsanitizer            x86_64   7.3.1-15.amzn2             amzn2-core   642 k
2022-11-23T01:16:55.9645093Z  m4                      x86_64   1.4.16-10.amzn2.0.2        amzn2-core   256 k
2022-11-23T01:16:55.9645514Z  mokutil                 x86_64   1:0.3.0-10.amzn2.0.1       amzn2-core    39 k
2022-11-23T01:16:55.9645921Z  mpfr                    x86_64   3.1.1-4.amzn2.0.2          amzn2-core   208 k
2022-11-23T01:16:55.9646341Z  neon                    x86_64   0.30.0-3.amzn2.0.2         amzn2-core   166 k
2022-11-23T01:16:55.9646774Z  pakchois                x86_64   0.4-10.amzn2.0.2           amzn2-core    14 k
2022-11-23T01:16:55.9647208Z  perl-Data-Dumper        x86_64   2.145-3.amzn2.0.2          amzn2-core    48 k
2022-11-23T01:16:55.9647676Z  perl-Test-Harness       noarch   3.28-3.amzn2               amzn2-core   302 k
2022-11-23T01:16:55.9648154Z  perl-Thread-Queue       noarch   3.02-2.amzn2               amzn2-core    17 k
2022-11-23T01:16:55.9648636Z  perl-XML-Parser         x86_64   2.41-10.amzn2.0.2          amzn2-core   223 k
2022-11-23T01:16:55.9649092Z  perl-srpm-macros        noarch   1-8.amzn2.0.1              amzn2-core   4.7 k
2022-11-23T01:16:55.9649563Z  subversion-libs         x86_64   1.7.14-16.amzn2.0.1        amzn2-core   912 k
2022-11-23T01:16:55.9650019Z  systemtap-client        x86_64   4.5-1.amzn2.0.1            amzn2-core   3.7 M
2022-11-23T01:16:55.9650458Z  systemtap-devel         x86_64   4.5-1.amzn2.0.1            amzn2-core   2.3 M
2022-11-23T01:16:55.9650905Z  trousers                x86_64   0.3.14-2.amzn2.0.2         amzn2-core   294 k
2022-11-23T01:16:55.9651337Z  zlib-devel              x86_64   1.2.7-19.amzn2.0.2         amzn2-core    50 k
2022-11-23T01:16:55.9651547Z 
2022-11-23T01:16:55.9651663Z Transaction Summary
2022-11-23T01:16:55.9651932Z ================================================================================
2022-11-23T01:16:55.9652257Z Install  25 Packages (+43 Dependent packages)
2022-11-23T01:16:55.9652459Z 
2022-11-23T01:16:55.9652587Z Total download size: 96 M
2022-11-23T01:16:55.9652836Z Installed size: 303 M
2022-11-23T01:16:55.9653099Z Downloading packages:
2022-11-23T01:16:55.9667852Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-11-23T01:16:57.2050007Z --------------------------------------------------------------------------------
2022-11-23T01:16:57.2050553Z Total                                               77 MB/s |  96 MB  00:01     
2022-11-23T01:16:57.3099760Z Running transaction check
2022-11-23T01:16:57.3862334Z Running transaction test
2022-11-23T01:16:57.7988951Z Transaction test succeeded
2022-11-23T01:16:57.7992202Z Running transaction
2022-11-23T01:17:03.1679862Z   Installing : mpfr-3.1.1-4.amzn2.0.2.x86_64                               1/68 
2022-11-23T01:17:05.6288521Z   Installing : libmpc-1.0.1-3.amzn2.0.2.x86_64                             2/68 
2022-11-23T01:17:08.0971727Z   Installing : m4-1.4.16-10.amzn2.0.2.x86_64                               3/68 
2022-11-23T01:17:10.5264037Z   Installing : apr-1.7.0-9.amzn2.x86_64                                    4/68 
2022-11-23T01:17:12.9476633Z   Installing : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                       5/68 
2022-11-23T01:17:15.3872524Z   Installing : apr-util-1.6.1-5.amzn2.0.2.x86_64                           6/68 
2022-11-23T01:17:16.0292337Z   Installing : avahi-libs-0.6.31-20.amzn2.x86_64                           7/68 
2022-11-23T01:17:16.0793875Z   Installing : libquadmath-7.3.1-15.amzn2.x86_64                           8/68 
2022-11-23T01:17:16.1106374Z   Installing : patch-2.7.1-12.amzn2.0.2.x86_64                             9/68 
2022-11-23T01:17:16.2013101Z   Installing : perl-Thread-Queue-3.02-2.amzn2.noarch                      10/68 
2022-11-23T01:17:17.2705244Z   Installing : libgfortran-7.3.1-15.amzn2.x86_64                          11/68 
2022-11-23T01:17:17.3494671Z   Installing : cpp-7.3.1-15.amzn2.x86_64                                  12/68 
2022-11-23T01:17:17.4026805Z   Installing : libmodman-2.0.1-8.amzn2.0.2.x86_64                         13/68 
2022-11-23T01:17:17.4763581Z   Installing : libproxy-0.4.11-10.amzn2.0.3.x86_64                        14/68 
2022-11-23T01:17:17.5421338Z   Installing : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   15/68 
2022-11-23T01:17:17.6621322Z   Installing : elfutils-0.176-2.amzn2.x86_64                              16/68 
2022-11-23T01:17:17.7015763Z   Installing : libsanitizer-7.3.1-15.amzn2.x86_64                         17/68 
2022-11-23T01:17:17.7303221Z   Installing : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 18/68 
2022-11-23T01:17:17.7635231Z   Installing : efivar-libs-31-4.amzn2.0.4.x86_64                          19/68 
2022-11-23T01:17:17.7964601Z   Installing : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        20/68 
2022-11-23T01:17:17.8786095Z   Installing : gettext-common-devel-0.19.8.1-3.amzn2.noarch               21/68 
2022-11-23T01:17:17.9325240Z   Installing : gettext-devel-0.19.8.1-3.amzn2.x86_64                      22/68 
2022-11-23T01:17:18.0254906Z   Installing : dwz-0.11-3.amzn2.0.3.x86_64                                23/68 
2022-11-23T01:17:18.2076844Z   Installing : trousers-0.3.14-2.amzn2.0.2.x86_64                         24/68 
2022-11-23T01:17:18.2505364Z   Installing : gnutls-3.3.29-9.amzn2.0.1.x86_64                           25/68 
2022-11-23T01:17:18.6520211Z   Installing : libitm-7.3.1-15.amzn2.x86_64                               26/68 
2022-11-23T01:17:18.6902326Z   Installing : gdb-8.0.1-36.amzn2.0.1.x86_64                              27/68 
2022-11-23T01:17:18.7194118Z   Installing : libmpx-7.3.1-15.amzn2.x86_64                               28/68 
2022-11-23T01:17:18.7399939Z   Installing : perl-srpm-macros-1-8.amzn2.0.1.noarch                      29/68 
2022-11-23T01:17:18.7743421Z   Installing : go-srpm-macros-3.0.15-23.amzn2.0.2.noarch                  30/68 
2022-11-23T01:17:18.8082533Z   Installing : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               31/68 
2022-11-23T01:17:18.9082342Z   Installing : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  32/68 
2022-11-23T01:17:19.0020125Z   Installing : autoconf-2.69-11.amzn2.noarch                              33/68 
2022-11-23T01:17:19.1105531Z   Installing : perl-Test-Harness-3.28-3.amzn2.noarch                      34/68 
2022-11-23T01:17:19.1645482Z   Installing : automake-1.13.4-3.1.amzn2.noarch                           35/68 
2022-11-23T01:17:19.2014454Z   Installing : zlib-devel-1.2.7-19.amzn2.0.2.x86_64                       36/68 
2022-11-23T01:17:19.2484621Z   Installing : elfutils-libelf-devel-0.176-2.amzn2.x86_64                 37/68 
2022-11-23T01:17:19.5846064Z   Installing : libatomic-7.3.1-15.amzn2.x86_64                            38/68 
2022-11-23T01:17:19.7600988Z   Installing : kernel-headers-4.14.296-222.539.amzn2.x86_64               39/68 
2022-11-23T01:17:19.8994742Z   Installing : glibc-headers-2.26-62.amzn2.x86_64                         40/68 
2022-11-23T01:17:19.9408018Z   Installing : glibc-devel-2.26-62.amzn2.x86_64                           41/68 
2022-11-23T01:17:22.0747935Z   Installing : libcilkrts-7.3.1-15.amzn2.x86_64                           42/68 
2022-11-23T01:17:26.0654463Z   Installing : gcc-7.3.1-15.amzn2.x86_64                                  43/68 
2022-11-23T01:17:37.2923984Z   Installing : kernel-devel-4.14.296-222.539.amzn2.x86_64                 44/68 
2022-11-23T01:17:37.9358239Z   Installing : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     45/68 
2022-11-23T01:17:38.0035537Z   Installing : systemtap-client-4.5-1.amzn2.0.1.x86_64                    46/68 
2022-11-23T01:17:38.0619978Z   Installing : pakchois-0.4-10.amzn2.0.2.x86_64                           47/68 
2022-11-23T01:17:38.1997782Z   Installing : neon-0.30.0-3.amzn2.0.2.x86_64                             48/68 
2022-11-23T01:17:38.3882090Z   Installing : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 49/68 
2022-11-23T01:17:38.4926378Z   Installing : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-11-23T01:17:39.7202741Z   Installing : systemtap-4.5-1.amzn2.0.1.x86_64                           51/68 
2022-11-23T01:17:41.3551049Z   Installing : gcc-gfortran-7.3.1-15.amzn2.x86_64                         52/68 
2022-11-23T01:17:41.4735097Z   Installing : gcc-c++-7.3.1-15.amzn2.x86_64                              53/68 
2022-11-23T01:17:41.5165706Z   Installing : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        54/68 
2022-11-23T01:17:41.5592338Z   Installing : intltool-0.50.2-7.amzn2.noarch                             55/68 
2022-11-23T01:17:41.6316641Z   Installing : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       56/68 
2022-11-23T01:17:41.6991124Z   Installing : cscope-15.8-10.amzn2.0.2.x86_64                            57/68 
2022-11-23T01:17:41.8073068Z   Installing : flex-2.5.37-3.amzn2.0.3.x86_64                             58/68 
2022-11-23T01:17:41.8760929Z   Installing : bison-3.0.4-6.amzn2.0.2.x86_64                             59/68 
2022-11-23T01:17:41.9253470Z   Installing : rcs-5.9.0-5.amzn2.0.2.x86_64                               60/68 
2022-11-23T01:17:41.9683168Z   Installing : ctags-5.8-13.amzn2.0.2.x86_64                              61/68 
2022-11-23T01:17:42.0175516Z   Installing : indent-2.2.11-13.amzn2.0.2.x86_64                          62/68 
2022-11-23T01:17:42.7312869Z   Installing : patchutils-0.3.3-4.amzn2.0.1.x86_64                        63/68 
2022-11-23T01:17:42.7791032Z   Installing : 1:doxygen-1.8.5-4.amzn2.x86_64                             64/68 
2022-11-23T01:17:42.8138484Z   Installing : diffstat-1.57-4.amzn2.0.2.x86_64                           65/68 
2022-11-23T01:17:43.1423136Z   Installing : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      66/68 
2022-11-23T01:17:43.1859019Z   Installing : swig-3.0.12-11.amzn2.0.3.x86_64                            67/68 
2022-11-23T01:17:43.2572075Z   Installing : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                        68/68 
2022-11-23T01:17:43.2717709Z   Verifying  : elfutils-libelf-devel-0.176-2.amzn2.x86_64                  1/68 
2022-11-23T01:17:43.2837020Z   Verifying  : perl-Thread-Queue-3.02-2.amzn2.noarch                       2/68 
2022-11-23T01:17:43.2981339Z   Verifying  : gettext-devel-0.19.8.1-3.amzn2.x86_64                       3/68 
2022-11-23T01:17:43.3094410Z   Verifying  : patch-2.7.1-12.amzn2.0.2.x86_64                             4/68 
2022-11-23T01:17:43.3270803Z   Verifying  : kernel-devel-4.14.296-222.539.amzn2.x86_64                  5/68 
2022-11-23T01:17:43.3424166Z   Verifying  : flex-2.5.37-3.amzn2.0.3.x86_64                              6/68 
2022-11-23T01:17:43.3637471Z   Verifying  : pakchois-0.4-10.amzn2.0.2.x86_64                            7/68 
2022-11-23T01:17:43.3791518Z   Verifying  : rpm-sign-4.11.3-48.amzn2.0.2.x86_64                         8/68 
2022-11-23T01:17:43.4140820Z   Verifying  : glibc-devel-2.26-62.amzn2.x86_64                            9/68 
2022-11-23T01:17:43.4289261Z   Verifying  : gcc-gfortran-7.3.1-15.amzn2.x86_64                         10/68 
2022-11-23T01:17:43.4450487Z   Verifying  : swig-3.0.12-11.amzn2.0.3.x86_64                            11/68 
2022-11-23T01:17:43.4605862Z   Verifying  : byacc-1.9.20130304-3.amzn2.0.2.x86_64                      12/68 
2022-11-23T01:17:43.4753286Z   Verifying  : libmpc-1.0.1-3.amzn2.0.2.x86_64                            13/68 
2022-11-23T01:17:43.4887224Z   Verifying  : libcilkrts-7.3.1-15.amzn2.x86_64                           14/68 
2022-11-23T01:17:43.5026789Z   Verifying  : kernel-headers-4.14.296-222.539.amzn2.x86_64               15/68 
2022-11-23T01:17:43.5138077Z   Verifying  : libproxy-0.4.11-10.amzn2.0.3.x86_64                        16/68 
2022-11-23T01:17:43.5270298Z   Verifying  : cscope-15.8-10.amzn2.0.2.x86_64                            17/68 
2022-11-23T01:17:43.5411920Z   Verifying  : diffstat-1.57-4.amzn2.0.2.x86_64                           18/68 
2022-11-23T01:17:43.5541680Z   Verifying  : 1:doxygen-1.8.5-4.amzn2.x86_64                             19/68 
2022-11-23T01:17:43.5683880Z   Verifying  : gcc-c++-7.3.1-15.amzn2.x86_64                              20/68 
2022-11-23T01:17:43.5780196Z   Verifying  : libatomic-7.3.1-15.amzn2.x86_64                            21/68 
2022-11-23T01:17:43.5892835Z   Verifying  : system-rpm-config-9.1.0-76.amzn2.0.14.noarch               22/68 
2022-11-23T01:17:43.6009287Z   Verifying  : systemtap-devel-4.5-1.amzn2.0.1.x86_64                     23/68 
2022-11-23T01:17:43.6105015Z   Verifying  : zlib-devel-1.2.7-19.amzn2.0.2.x86_64                       24/68 
2022-11-23T01:17:43.6215058Z   Verifying  : glibc-headers-2.26-62.amzn2.x86_64                         25/68 
2022-11-23T01:17:43.6330163Z   Verifying  : perl-Test-Harness-3.28-3.amzn2.noarch                      26/68 
2022-11-23T01:17:43.6440026Z   Verifying  : autoconf-2.69-11.amzn2.noarch                              27/68 
2022-11-23T01:17:43.6554653Z   Verifying  : libquadmath-7.3.1-15.amzn2.x86_64                          28/68 
2022-11-23T01:17:43.6675323Z   Verifying  : intltool-0.50.2-7.amzn2.noarch                             29/68 
2022-11-23T01:17:43.6786368Z   Verifying  : apr-util-1.6.1-5.amzn2.0.2.x86_64                          30/68 
2022-11-23T01:17:43.6896748Z   Verifying  : cpp-7.3.1-15.amzn2.x86_64                                  31/68 
2022-11-23T01:17:43.7012187Z   Verifying  : rpm-build-4.11.3-48.amzn2.0.2.x86_64                       32/68 
2022-11-23T01:17:43.7138483Z   Verifying  : go-srpm-macros-3.0.15-23.amzn2.0.2.noarch                  33/68 
2022-11-23T01:17:43.7250269Z   Verifying  : perl-Data-Dumper-2.145-3.amzn2.0.2.x86_64                  34/68 
2022-11-23T01:17:43.7387670Z   Verifying  : perl-srpm-macros-1-8.amzn2.0.1.noarch                      35/68 
2022-11-23T01:17:43.7509403Z   Verifying  : gnutls-3.3.29-9.amzn2.0.1.x86_64                           36/68 
2022-11-23T01:17:43.7635081Z   Verifying  : subversion-libs-1.7.14-16.amzn2.0.1.x86_64                 37/68 
2022-11-23T01:17:43.7745837Z   Verifying  : automake-1.13.4-3.1.amzn2.noarch                           38/68 
2022-11-23T01:17:43.7857375Z   Verifying  : apr-util-bdb-1.6.1-5.amzn2.0.2.x86_64                      39/68 
2022-11-23T01:17:43.7981575Z   Verifying  : libmpx-7.3.1-15.amzn2.x86_64                               40/68 
2022-11-23T01:17:43.8099880Z   Verifying  : avahi-libs-0.6.31-20.amzn2.x86_64                          41/68 
2022-11-23T01:17:43.8233242Z   Verifying  : bison-3.0.4-6.amzn2.0.2.x86_64                             42/68 
2022-11-23T01:17:43.8360470Z   Verifying  : libgfortran-7.3.1-15.amzn2.x86_64                          43/68 
2022-11-23T01:17:43.8644374Z   Verifying  : gdb-8.0.1-36.amzn2.0.1.x86_64                              44/68 
2022-11-23T01:17:43.8758379Z   Verifying  : patchutils-0.3.3-4.amzn2.0.1.x86_64                        45/68 
2022-11-23T01:17:43.8886180Z   Verifying  : libitm-7.3.1-15.amzn2.x86_64                               46/68 
2022-11-23T01:17:43.9006606Z   Verifying  : libtool-2.4.2-22.2.amzn2.0.2.x86_64                        47/68 
2022-11-23T01:17:43.9121857Z   Verifying  : gcc-7.3.1-15.amzn2.x86_64                                  48/68 
2022-11-23T01:17:43.9220855Z   Verifying  : indent-2.2.11-13.amzn2.0.2.x86_64                          49/68 
2022-11-23T01:17:43.9341598Z   Verifying  : subversion-1.7.14-16.amzn2.0.1.x86_64                      50/68 
2022-11-23T01:17:43.9493845Z   Verifying  : apr-1.7.0-9.amzn2.x86_64                                   51/68 
2022-11-23T01:17:43.9620657Z   Verifying  : ctags-5.8-13.amzn2.0.2.x86_64                              52/68 
2022-11-23T01:17:43.9752606Z   Verifying  : 1:mokutil-0.3.0-10.amzn2.0.1.x86_64                        53/68 
2022-11-23T01:17:43.9865234Z   Verifying  : mpfr-3.1.1-4.amzn2.0.2.x86_64                              54/68 
2022-11-23T01:17:43.9960333Z   Verifying  : trousers-0.3.14-2.amzn2.0.2.x86_64                         55/68 
2022-11-23T01:17:44.0087753Z   Verifying  : neon-0.30.0-3.amzn2.0.2.x86_64                             56/68 
2022-11-23T01:17:44.0233115Z   Verifying  : systemtap-4.5-1.amzn2.0.1.x86_64                           57/68 
2022-11-23T01:17:44.0374818Z   Verifying  : dwz-0.11-3.amzn2.0.3.x86_64                                58/68 
2022-11-23T01:17:44.0504170Z   Verifying  : gettext-common-devel-0.19.8.1-3.amzn2.noarch               59/68 
2022-11-23T01:17:44.0629587Z   Verifying  : systemtap-client-4.5-1.amzn2.0.1.x86_64                    60/68 
2022-11-23T01:17:44.0750532Z   Verifying  : efivar-libs-31-4.amzn2.0.4.x86_64                          61/68 
2022-11-23T01:17:44.0870329Z   Verifying  : rcs-5.9.0-5.amzn2.0.2.x86_64                               62/68 
2022-11-23T01:17:44.0994659Z   Verifying  : 1:emacs-filesystem-27.2-4.amzn2.0.1.noarch                 63/68 
2022-11-23T01:17:44.1127511Z   Verifying  : libsanitizer-7.3.1-15.amzn2.x86_64                         64/68 
2022-11-23T01:17:44.1286898Z   Verifying  : elfutils-0.176-2.amzn2.x86_64                              65/68 
2022-11-23T01:17:44.1404391Z   Verifying  : m4-1.4.16-10.amzn2.0.2.x86_64                              66/68 
2022-11-23T01:17:44.1515950Z   Verifying  : perl-XML-Parser-2.41-10.amzn2.0.2.x86_64                   67/68 
2022-11-23T01:17:44.2305845Z   Verifying  : libmodman-2.0.1-8.amzn2.0.2.x86_64                         68/68 
2022-11-23T01:17:44.2307829Z 
2022-11-23T01:17:44.2308557Z Installed:
2022-11-23T01:17:44.2309172Z   autoconf.noarch 0:2.69-11.amzn2                                               
2022-11-23T01:17:44.2309754Z   automake.noarch 0:1.13.4-3.1.amzn2                                            
2022-11-23T01:17:44.2314751Z   bison.x86_64 0:3.0.4-6.amzn2.0.2                                              
2022-11-23T01:17:44.2315210Z   byacc.x86_64 0:1.9.20130304-3.amzn2.0.2                                       
2022-11-23T01:17:44.2315656Z   cscope.x86_64 0:15.8-10.amzn2.0.2                                             
2022-11-23T01:17:44.2316087Z   ctags.x86_64 0:5.8-13.amzn2.0.2                                               
2022-11-23T01:17:44.2316522Z   diffstat.x86_64 0:1.57-4.amzn2.0.2                                            
2022-11-23T01:17:44.2316941Z   doxygen.x86_64 1:1.8.5-4.amzn2                                                
2022-11-23T01:17:44.2317364Z   elfutils.x86_64 0:0.176-2.amzn2                                               
2022-11-23T01:17:44.2317762Z   flex.x86_64 0:2.5.37-3.amzn2.0.3                                              
2022-11-23T01:17:44.2318172Z   gcc.x86_64 0:7.3.1-15.amzn2                                                   
2022-11-23T01:17:44.2318585Z   gcc-c++.x86_64 0:7.3.1-15.amzn2                                               
2022-11-23T01:17:44.2318990Z   gcc-gfortran.x86_64 0:7.3.1-15.amzn2                                          
2022-11-23T01:17:44.2320295Z   indent.x86_64 0:2.2.11-13.amzn2.0.2                                           
2022-11-23T01:17:44.2320737Z   intltool.noarch 0:0.50.2-7.amzn2                                              
2022-11-23T01:17:44.2321455Z   libtool.x86_64 0:2.4.2-22.2.amzn2.0.2                                         
2022-11-23T01:17:44.2321913Z   patch.x86_64 0:2.7.1-12.amzn2.0.2                                             
2022-11-23T01:17:44.2322338Z   patchutils.x86_64 0:0.3.3-4.amzn2.0.1                                         
2022-11-23T01:17:44.2322751Z   rcs.x86_64 0:5.9.0-5.amzn2.0.2                                                
2022-11-23T01:17:44.2323491Z   rpm-build.x86_64 0:4.11.3-48.amzn2.0.2                                        
2022-11-23T01:17:44.2324259Z   rpm-sign.x86_64 0:4.11.3-48.amzn2.0.2                                         
2022-11-23T01:17:44.2325216Z   subversion.x86_64 0:1.7.14-16.amzn2.0.1                                       
2022-11-23T01:17:44.2326060Z   swig.x86_64 0:3.0.12-11.amzn2.0.3                                             
2022-11-23T01:17:44.2326994Z   system-rpm-config.noarch 0:9.1.0-76.amzn2.0.14                                
2022-11-23T01:17:44.2327930Z   systemtap.x86_64 0:4.5-1.amzn2.0.1                                            
2022-11-23T01:17:44.2328170Z 
2022-11-23T01:17:44.2328293Z Dependency Installed:
2022-11-23T01:17:44.2328694Z   apr.x86_64 0:1.7.0-9.amzn2                                                    
2022-11-23T01:17:44.2329100Z   apr-util.x86_64 0:1.6.1-5.amzn2.0.2                                           
2022-11-23T01:17:44.2329539Z   apr-util-bdb.x86_64 0:1.6.1-5.amzn2.0.2                                       
2022-11-23T01:17:44.2329973Z   avahi-libs.x86_64 0:0.6.31-20.amzn2                                           
2022-11-23T01:17:44.2330393Z   cpp.x86_64 0:7.3.1-15.amzn2                                                   
2022-11-23T01:17:44.2330785Z   dwz.x86_64 0:0.11-3.amzn2.0.3                                                 
2022-11-23T01:17:44.2331203Z   efivar-libs.x86_64 0:31-4.amzn2.0.4                                           
2022-11-23T01:17:44.2331666Z   elfutils-libelf-devel.x86_64 0:0.176-2.amzn2                                  
2022-11-23T01:17:44.2332117Z   emacs-filesystem.noarch 1:27.2-4.amzn2.0.1                                    
2022-11-23T01:17:44.2332551Z   gdb.x86_64 0:8.0.1-36.amzn2.0.1                                               
2022-11-23T01:17:44.2332994Z   gettext-common-devel.noarch 0:0.19.8.1-3.amzn2                                
2022-11-23T01:17:44.2333449Z   gettext-devel.x86_64 0:0.19.8.1-3.amzn2                                       
2022-11-23T01:17:44.2333863Z   glibc-devel.x86_64 0:2.26-62.amzn2                                            
2022-11-23T01:17:44.2334295Z   glibc-headers.x86_64 0:2.26-62.amzn2                                          
2022-11-23T01:17:44.2334719Z   gnutls.x86_64 0:3.3.29-9.amzn2.0.1                                            
2022-11-23T01:17:44.2335132Z   go-srpm-macros.noarch 0:3.0.15-23.amzn2.0.2                                   
2022-11-23T01:17:44.2335575Z   kernel-devel.x86_64 0:4.14.296-222.539.amzn2                                  
2022-11-23T01:17:44.2336019Z   kernel-headers.x86_64 0:4.14.296-222.539.amzn2                                
2022-11-23T01:17:44.2336452Z   libatomic.x86_64 0:7.3.1-15.amzn2                                             
2022-11-23T01:17:44.2336859Z   libcilkrts.x86_64 0:7.3.1-15.amzn2                                            
2022-11-23T01:17:44.2337282Z   libgfortran.x86_64 0:7.3.1-15.amzn2                                           
2022-11-23T01:17:44.2337701Z   libitm.x86_64 0:7.3.1-15.amzn2                                                
2022-11-23T01:17:44.2338097Z   libmodman.x86_64 0:2.0.1-8.amzn2.0.2                                          
2022-11-23T01:17:44.2338518Z   libmpc.x86_64 0:1.0.1-3.amzn2.0.2                                             
2022-11-23T01:17:44.2338926Z   libmpx.x86_64 0:7.3.1-15.amzn2                                                
2022-11-23T01:17:44.2339336Z   libproxy.x86_64 0:0.4.11-10.amzn2.0.3                                         
2022-11-23T01:17:44.2339921Z   libquadmath.x86_64 0:7.3.1-15.amzn2                                           
2022-11-23T01:17:44.2340359Z   libsanitizer.x86_64 0:7.3.1-15.amzn2                                          
2022-11-23T01:17:44.2340774Z   m4.x86_64 0:1.4.16-10.amzn2.0.2                                               
2022-11-23T01:17:44.2341162Z   mokutil.x86_64 1:0.3.0-10.amzn2.0.1                                           
2022-11-23T01:17:44.2341576Z   mpfr.x86_64 0:3.1.1-4.amzn2.0.2                                               
2022-11-23T01:17:44.2341985Z   neon.x86_64 0:0.30.0-3.amzn2.0.2                                              
2022-11-23T01:17:44.2342402Z   pakchois.x86_64 0:0.4-10.amzn2.0.2                                            
2022-11-23T01:17:44.2342826Z   perl-Data-Dumper.x86_64 0:2.145-3.amzn2.0.2                                   
2022-11-23T01:17:44.2343286Z   perl-Test-Harness.noarch 0:3.28-3.amzn2                                       
2022-11-23T01:17:44.2343767Z   perl-Thread-Queue.noarch 0:3.02-2.amzn2                                       
2022-11-23T01:17:44.2344234Z   perl-XML-Parser.x86_64 0:2.41-10.amzn2.0.2                                    
2022-11-23T01:17:44.2344700Z   perl-srpm-macros.noarch 0:1-8.amzn2.0.1                                       
2022-11-23T01:17:44.2345156Z   subversion-libs.x86_64 0:1.7.14-16.amzn2.0.1                                  
2022-11-23T01:17:44.2345597Z   systemtap-client.x86_64 0:4.5-1.amzn2.0.1                                     
2022-11-23T01:17:44.2346027Z   systemtap-devel.x86_64 0:4.5-1.amzn2.0.1                                      
2022-11-23T01:17:44.2346455Z   trousers.x86_64 0:0.3.14-2.amzn2.0.2                                          
2022-11-23T01:17:44.2347520Z   zlib-devel.x86_64 0:1.2.7-19.amzn2.0.2                                        
2022-11-23T01:17:44.2347755Z 
2022-11-23T01:17:44.2347845Z Complete!
2022-11-23T01:17:44.2697912Z ++ uname -r
2022-11-23T01:17:44.2706786Z + sudo yum install -y 'kernel-devel-uname-r == 4.14.252-195.483.amzn2.x86_64'
2022-11-23T01:17:44.7760055Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:17:45.0601556Z Resolving Dependencies
2022-11-23T01:17:45.0608153Z --> Running transaction check
2022-11-23T01:17:45.0608940Z ---> Package kernel-devel.x86_64 0:4.14.252-195.483.amzn2 will be installed
2022-11-23T01:17:45.3447716Z --> Finished Dependency Resolution
2022-11-23T01:17:45.4259172Z 
2022-11-23T01:17:45.4259829Z Dependencies Resolved
2022-11-23T01:17:45.4265581Z 
2022-11-23T01:17:45.4265949Z ================================================================================
2022-11-23T01:17:45.4266598Z  Package          Arch       Version                       Repository      Size
2022-11-23T01:17:45.4267569Z ================================================================================
2022-11-23T01:17:45.4267857Z Installing:
2022-11-23T01:17:45.4268389Z  kernel-devel     x86_64     4.14.252-195.483.amzn2        amzn2-core      13 M
2022-11-23T01:17:45.4268629Z 
2022-11-23T01:17:45.4268768Z Transaction Summary
2022-11-23T01:17:45.4269041Z ================================================================================
2022-11-23T01:17:45.4269321Z Install  1 Package
2022-11-23T01:17:45.4269482Z 
2022-11-23T01:17:45.4269603Z Total download size: 13 M
2022-11-23T01:17:45.4269852Z Installed size: 60 M
2022-11-23T01:17:45.4270284Z Downloading packages:
2022-11-23T01:17:45.4279774Z Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2022-11-23T01:17:45.7212676Z Running transaction check
2022-11-23T01:17:45.7395547Z Running transaction test
2022-11-23T01:17:46.1435587Z Transaction test succeeded
2022-11-23T01:17:46.1438371Z Running transaction
2022-11-23T01:18:01.6000335Z   Installing : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-11-23T01:18:01.6902016Z   Verifying  : kernel-devel-4.14.252-195.483.amzn2.x86_64                   1/1 
2022-11-23T01:18:01.6902551Z 
2022-11-23T01:18:01.6902780Z Installed:
2022-11-23T01:18:01.6903825Z   kernel-devel.x86_64 0:4.14.252-195.483.amzn2                                  
2022-11-23T01:18:01.6904058Z 
2022-11-23T01:18:01.6904150Z Complete!
2022-11-23T01:18:01.7234132Z + sudo modprobe backlight
2022-11-23T01:18:01.7542171Z + sudo curl -fsL -o /tmp/nvidia_driver https://s3.amazonaws.com/ossci-linux/nvidia_driver/NVIDIA-Linux-x86_64-515.76.run
2022-11-23T01:18:05.4342896Z + set +e
2022-11-23T01:18:05.4343824Z + sudo /bin/bash /tmp/nvidia_driver -s --no-drm
2022-11-23T01:18:06.8160200Z Verifying archive integrity... OK
2022-11-23T01:18:33.6380142Z Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 515.76................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2022-11-23T01:18:33.7799147Z 
2022-11-23T01:18:33.7803455Z WARNING: The nvidia-drm module will not be installed. As a result, DRM-KMS will not function with this installation of the NVIDIA driver.
2022-11-23T01:18:33.7804183Z 
2022-11-23T01:18:46.8644556Z 
2022-11-23T01:18:46.8646602Z WARNING: nvidia-installer was forced to guess the X library path '/usr/lib64' and X module path '/usr/lib64/xorg/modules'; these paths were not queryable from the system.  If X fails to find the NVIDIA X driver module, please install the `pkg-config` utility and the X.Org SDK/development package for your distribution and reinstall the driver.
2022-11-23T01:18:46.8647752Z 
2022-11-23T01:18:57.4518448Z + NVIDIA_INSTALLATION_STATUS=0
2022-11-23T01:18:57.4518790Z + RESET_GPU=0
2022-11-23T01:18:57.4519177Z + '[' 0 -ne 0 ']'
2022-11-23T01:18:57.4521083Z ++ command -v nvidia-smi
2022-11-23T01:18:57.4523522Z + '[' -x /usr/bin/nvidia-smi ']'
2022-11-23T01:18:57.4527176Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2022-11-23T01:19:02.5487066Z + INSTALLED_DRIVER_VERSION=515.76
2022-11-23T01:19:02.5487395Z + NVIDIA_SMI_STATUS=0
2022-11-23T01:19:02.5487808Z + '[' 0 -ne 0 ']'
2022-11-23T01:19:02.5488082Z + '[' 0 -eq 1 ']'
2022-11-23T01:19:02.5488383Z + sudo rm -fv /tmp/nvidia_driver
2022-11-23T01:19:02.6127436Z removed ‘/tmp/nvidia_driver’
2022-11-23T01:19:02.6143494Z + set -e
2022-11-23T01:19:02.6143839Z + sudo modprobe nvidia
2022-11-23T01:19:02.6258895Z + echo 'After installing NVIDIA driver'
2022-11-23T01:19:02.6259188Z + lspci
2022-11-23T01:19:02.6259426Z After installing NVIDIA driver
2022-11-23T01:19:02.6499945Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2022-11-23T01:19:02.6500373Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2022-11-23T01:19:02.6500788Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2022-11-23T01:19:02.6501169Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2022-11-23T01:19:02.6501545Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2022-11-23T01:19:02.6502561Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2022-11-23T01:19:02.6502992Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:19:02.6503399Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2022-11-23T01:19:02.6507673Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2022-11-23T01:19:02.6508076Z + lsmod
2022-11-23T01:19:02.6523037Z Module                  Size  Used by
2022-11-23T01:19:02.6523359Z nvidia              40808448  0
2022-11-23T01:19:02.6523607Z drm                   425984  1 nvidia
2022-11-23T01:19:02.6524104Z i2c_core               77824  2 nvidia,drm
2022-11-23T01:19:02.6524544Z backlight              16384  0
2022-11-23T01:19:02.6524803Z xt_conntrack           16384  1
2022-11-23T01:19:02.6525075Z ipt_MASQUERADE         16384  1
2022-11-23T01:19:02.6525377Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2022-11-23T01:19:02.6525683Z nf_conntrack_netlink    49152  0
2022-11-23T01:19:02.6525967Z nfnetlink              16384  2 nf_conntrack_netlink
2022-11-23T01:19:02.6526252Z xfrm_user              45056  1
2022-11-23T01:19:02.6526520Z xfrm_algo              16384  1 xfrm_user
2022-11-23T01:19:02.6526784Z xt_addrtype            16384  2
2022-11-23T01:19:02.6527064Z iptable_filter         16384  1
2022-11-23T01:19:02.6527331Z iptable_nat            16384  1
2022-11-23T01:19:02.6527583Z nf_conntrack_ipv4      16384  3
2022-11-23T01:19:02.6527895Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2022-11-23T01:19:02.6528205Z nf_nat_ipv4            16384  1 iptable_nat
2022-11-23T01:19:02.6528513Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2022-11-23T01:19:02.6528990Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2022-11-23T01:19:02.6529401Z br_netfilter           24576  0
2022-11-23T01:19:02.6529681Z bridge                172032  1 br_netfilter
2022-11-23T01:19:02.6531406Z stp                    16384  1 bridge
2022-11-23T01:19:02.6531710Z llc                    16384  2 bridge,stp
2022-11-23T01:19:02.6532001Z overlay                86016  0
2022-11-23T01:19:02.6532266Z sunrpc                393216  1
2022-11-23T01:19:02.6532508Z dm_mirror              28672  0
2022-11-23T01:19:02.6532797Z dm_region_hash         20480  1 dm_mirror
2022-11-23T01:19:02.6533109Z dm_log                 20480  2 dm_region_hash,dm_mirror
2022-11-23T01:19:02.6533407Z dm_mod                143360  2 dm_log,dm_mirror
2022-11-23T01:19:02.6533680Z dax                    69632  1 dm_mod
2022-11-23T01:19:02.6533941Z sb_edac                24576  0
2022-11-23T01:19:02.6534180Z crc32_pclmul           16384  0
2022-11-23T01:19:02.6534446Z ghash_clmulni_intel    16384  0
2022-11-23T01:19:02.6534717Z pcbc                   16384  0
2022-11-23T01:19:02.6534960Z aesni_intel           188416  0
2022-11-23T01:19:02.6535218Z ata_piix               36864  0
2022-11-23T01:19:02.6535483Z aes_x86_64             20480  1 aesni_intel
2022-11-23T01:19:02.6535757Z libata                266240  1 ata_piix
2022-11-23T01:19:02.6536023Z crypto_simd            16384  1 aesni_intel
2022-11-23T01:19:02.6536327Z glue_helper            16384  1 aesni_intel
2022-11-23T01:19:02.6536597Z mousedev               24576  0
2022-11-23T01:19:02.6536917Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2022-11-23T01:19:02.6537247Z pcc_cpufreq            16384  0
2022-11-23T01:19:02.6537515Z scsi_mod              245760  1 libata
2022-11-23T01:19:02.6537759Z psmouse                32768  0
2022-11-23T01:19:02.6538011Z evdev                  20480  3
2022-11-23T01:19:02.6538265Z button                 16384  0
2022-11-23T01:19:02.6538492Z ena                   114688  0
2022-11-23T01:19:02.6538772Z xen_blkfront           49152  2
2022-11-23T01:19:02.6539028Z crc32c_intel           24576  0
2022-11-23T01:19:02.6539284Z autofs4                49152  2
2022-11-23T01:19:02.6539514Z + modinfo nvidia
2022-11-23T01:19:02.6540009Z filename:       /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2022-11-23T01:19:02.6540362Z firmware:       nvidia/515.76/gsp.bin
2022-11-23T01:19:02.6540682Z alias:          char-major-195-*
2022-11-23T01:19:02.6540950Z version:        515.76
2022-11-23T01:19:02.6541355Z supported:      external
2022-11-23T01:19:02.6541657Z license:        NVIDIA
2022-11-23T01:19:02.6541938Z srcversion:     51FD9DD90150B35351AFFBB
2022-11-23T01:19:02.6542250Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2022-11-23T01:19:02.6542544Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2022-11-23T01:19:02.6542852Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2022-11-23T01:19:02.6543192Z depends:        i2c-core,drm
2022-11-23T01:19:02.6543440Z retpoline:      Y
2022-11-23T01:19:02.6543682Z name:           nvidia
2022-11-23T01:19:02.6544079Z vermagic:       4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 
2022-11-23T01:19:02.6544454Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2022-11-23T01:19:02.6544832Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2022-11-23T01:19:02.6545198Z parm:           NVreg_ResmanDebugLevel:int
2022-11-23T01:19:02.6545495Z parm:           NVreg_RmLogonRC:int
2022-11-23T01:19:02.6545788Z parm:           NVreg_ModifyDeviceFiles:int
2022-11-23T01:19:02.6546098Z parm:           NVreg_DeviceFileUID:int
2022-11-23T01:19:02.6546394Z parm:           NVreg_DeviceFileGID:int
2022-11-23T01:19:02.6546679Z parm:           NVreg_DeviceFileMode:int
2022-11-23T01:19:02.6547715Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2022-11-23T01:19:02.6548095Z parm:           NVreg_UsePageAttributeTable:int
2022-11-23T01:19:02.6548500Z parm:           NVreg_EnablePCIeGen3:int
2022-11-23T01:19:02.6548792Z parm:           NVreg_EnableMSI:int
2022-11-23T01:19:02.6549082Z parm:           NVreg_TCEBypassMode:int
2022-11-23T01:19:02.6549381Z parm:           NVreg_EnableStreamMemOPs:int
2022-11-23T01:19:02.6549744Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2022-11-23T01:19:02.6550135Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2022-11-23T01:19:02.6550511Z parm:           NVreg_EnableS0ixPowerManagement:int
2022-11-23T01:19:02.6550895Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2022-11-23T01:19:02.6551303Z parm:           NVreg_DynamicPowerManagement:int
2022-11-23T01:19:02.6551717Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2022-11-23T01:19:02.6552093Z parm:           NVreg_EnableGpuFirmware:int
2022-11-23T01:19:02.6552421Z parm:           NVreg_EnableGpuFirmwareLogs:int
2022-11-23T01:19:02.6552780Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2022-11-23T01:19:02.6553130Z parm:           NVreg_EnableUserNUMAManagement:int
2022-11-23T01:19:02.6553460Z parm:           NVreg_MemoryPoolSize:int
2022-11-23T01:19:02.6553778Z parm:           NVreg_KMallocHeapMaxSize:int
2022-11-23T01:19:02.6554085Z parm:           NVreg_VMallocHeapMaxSize:int
2022-11-23T01:19:02.6554398Z parm:           NVreg_IgnoreMMIOCheck:int
2022-11-23T01:19:02.6554696Z parm:           NVreg_NvLinkDisable:int
2022-11-23T01:19:02.6555042Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2022-11-23T01:19:02.6555378Z parm:           NVreg_RegisterPCIDriver:int
2022-11-23T01:19:02.6555709Z parm:           NVreg_EnableDbgBreakpoint:int
2022-11-23T01:19:02.6556033Z parm:           NVreg_RegistryDwords:charp
2022-11-23T01:19:02.6556355Z parm:           NVreg_RegistryDwordsPerDevice:charp
2022-11-23T01:19:02.6556670Z parm:           NVreg_RmMsg:charp
2022-11-23T01:19:02.6556961Z parm:           NVreg_GpuBlacklist:charp
2022-11-23T01:19:02.6557279Z parm:           NVreg_TemporaryFilePath:charp
2022-11-23T01:19:02.6557593Z parm:           NVreg_ExcludedGpus:charp
2022-11-23T01:19:02.6557906Z parm:           NVreg_DmaRemapPeerMmio:int
2022-11-23T01:19:02.6558193Z parm:           rm_firmware_active:charp
2022-11-23T01:19:02.6558454Z + set +e
2022-11-23T01:19:02.6558734Z + nvidia-smi
2022-11-23T01:19:06.4422440Z Wed Nov 23 01:19:06 2022       
2022-11-23T01:19:06.4423024Z +-----------------------------------------------------------------------------+
2022-11-23T01:19:06.4423598Z | NVIDIA-SMI 515.76       Driver Version: 515.76       CUDA Version: 11.7     |
2022-11-23T01:19:06.4424393Z |-------------------------------+----------------------+----------------------+
2022-11-23T01:19:06.4424932Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-11-23T01:19:06.4425420Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-11-23T01:19:06.4425785Z |                               |                      |               MIG M. |
2022-11-23T01:19:06.4426083Z |===============================+======================+======================|
2022-11-23T01:19:06.4469516Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |                    0 |
2022-11-23T01:19:06.4469880Z | N/A   27C    P0    37W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-11-23T01:19:06.4470197Z |                               |                      |                  N/A |
2022-11-23T01:19:06.4470666Z +-------------------------------+----------------------+----------------------+
2022-11-23T01:19:06.4516042Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |           2386512392 |
2022-11-23T01:19:06.4516403Z | N/A   32C    P0    37W / 150W |      0MiB /  7680MiB |     98%      Default |
2022-11-23T01:19:06.4516722Z |                               |                      |                  N/A |
2022-11-23T01:19:06.4517181Z +-------------------------------+----------------------+----------------------+
2022-11-23T01:19:06.4517538Z                                                                                
2022-11-23T01:19:06.4517967Z +-----------------------------------------------------------------------------+
2022-11-23T01:19:06.4518343Z | Processes:                                                                  |
2022-11-23T01:19:06.4518675Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-11-23T01:19:06.4519016Z |        ID   ID                                                   Usage      |
2022-11-23T01:19:06.4519311Z |=============================================================================|
2022-11-23T01:19:06.4521851Z |  No running processes found                                                 |
2022-11-23T01:19:06.4522329Z +-----------------------------------------------------------------------------+
2022-11-23T01:19:06.9876253Z + NVIDIA_SMI_STATUS=0
2022-11-23T01:19:06.9876622Z + '[' 0 -eq 0 ']'
2022-11-23T01:19:06.9876962Z + echo 'INFO: Ignoring allowed status 0'
2022-11-23T01:19:06.9877238Z + set -e
2022-11-23T01:19:06.9877488Z INFO: Ignoring allowed status 0
2022-11-23T01:19:06.9881796Z == Installing nvidia container toolkit for amzn2 ==
2022-11-23T01:19:06.9885821Z + sudo yum install -y yum-utils
2022-11-23T01:19:07.5394116Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:19:07.8110050Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-11-23T01:19:07.8110424Z Nothing to do
2022-11-23T01:19:07.8311908Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-11-23T01:19:08.3648276Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:19:08.3917719Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-11-23T01:19:08.3918394Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-11-23T01:19:08.3918907Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-11-23T01:19:08.4061727Z + sudo yum install -y nvidia-docker2
2022-11-23T01:19:08.9300221Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-11-23T01:19:08.9752535Z Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey
2022-11-23T01:19:08.9847028Z Importing GPG key 0xF796ECB0:
2022-11-23T01:19:08.9847399Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-11-23T01:19:08.9847799Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-11-23T01:19:08.9848275Z  From       : https://nvidia.github.io/libnvidia-container/gpgkey
2022-11-23T01:19:09.3796340Z Retrieving key from https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-11-23T01:19:09.3881274Z Importing GPG key 0xF796ECB0:
2022-11-23T01:19:09.3881697Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-11-23T01:19:09.3882078Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-11-23T01:19:09.3882608Z  From       : https://nvidia.github.io/nvidia-container-runtime/gpgkey
2022-11-23T01:19:09.6188270Z Retrieving key from https://nvidia.github.io/nvidia-docker/gpgkey
2022-11-23T01:19:09.6270353Z Importing GPG key 0xF796ECB0:
2022-11-23T01:19:09.6270748Z  Userid     : "NVIDIA CORPORATION (Open Source Projects) <cudatools@nvidia.com>"
2022-11-23T01:19:09.6271154Z  Fingerprint: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
2022-11-23T01:19:09.6271606Z  From       : https://nvidia.github.io/nvidia-docker/gpgkey
2022-11-23T01:19:11.4301640Z Resolving Dependencies
2022-11-23T01:19:11.4308618Z --> Running transaction check
2022-11-23T01:19:11.4309102Z ---> Package nvidia-docker2.noarch 0:2.11.0-1 will be installed
2022-11-23T01:19:11.4334117Z --> Processing Dependency: nvidia-container-toolkit >= 1.10.0-1 for package: nvidia-docker2-2.11.0-1.noarch
2022-11-23T01:19:11.4714994Z --> Running transaction check
2022-11-23T01:19:11.4715475Z ---> Package nvidia-container-toolkit.x86_64 0:1.11.0-1 will be installed
2022-11-23T01:19:11.4872156Z --> Processing Dependency: nvidia-container-toolkit-base = 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-11-23T01:19:11.4882567Z --> Processing Dependency: libnvidia-container-tools < 2.0.0 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-11-23T01:19:11.5012198Z --> Processing Dependency: libnvidia-container-tools >= 1.11.0-1 for package: nvidia-container-toolkit-1.11.0-1.x86_64
2022-11-23T01:19:11.5012693Z --> Running transaction check
2022-11-23T01:19:11.5013741Z ---> Package libnvidia-container-tools.x86_64 0:1.11.0-1 will be installed
2022-11-23T01:19:11.5023949Z --> Processing Dependency: libnvidia-container1(x86-64) >= 1.11.0-1 for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-11-23T01:19:11.5051295Z --> Processing Dependency: libnvidia-container.so.1(NVC_1.0)(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-11-23T01:19:11.5052028Z --> Processing Dependency: libnvidia-container.so.1()(64bit) for package: libnvidia-container-tools-1.11.0-1.x86_64
2022-11-23T01:19:11.5052914Z ---> Package nvidia-container-toolkit-base.x86_64 0:1.11.0-1 will be installed
2022-11-23T01:19:11.5055269Z --> Running transaction check
2022-11-23T01:19:11.5055994Z ---> Package libnvidia-container1.x86_64 0:1.11.0-1 will be installed
2022-11-23T01:19:11.7968005Z --> Finished Dependency Resolution
2022-11-23T01:19:11.8711193Z 
2022-11-23T01:19:11.8711530Z Dependencies Resolved
2022-11-23T01:19:11.8724584Z 
2022-11-23T01:19:11.8725055Z ================================================================================
2022-11-23T01:19:11.8725459Z  Package                        Arch    Version      Repository            Size
2022-11-23T01:19:11.8725808Z ================================================================================
2022-11-23T01:19:11.8726094Z Installing:
2022-11-23T01:19:11.8726604Z  nvidia-docker2                 noarch  2.11.0-1     libnvidia-container  8.7 k
2022-11-23T01:19:11.8726955Z Installing for dependencies:
2022-11-23T01:19:11.8727416Z  libnvidia-container-tools      x86_64  1.11.0-1     libnvidia-container   49 k
2022-11-23T01:19:11.8727909Z  libnvidia-container1           x86_64  1.11.0-1     libnvidia-container  1.0 M
2022-11-23T01:19:11.8728416Z  nvidia-container-toolkit       x86_64  1.11.0-1     libnvidia-container  780 k
2022-11-23T01:19:11.8728938Z  nvidia-container-toolkit-base  x86_64  1.11.0-1     libnvidia-container  2.5 M
2022-11-23T01:19:11.8729199Z 
2022-11-23T01:19:11.8729315Z Transaction Summary
2022-11-23T01:19:11.8729583Z ================================================================================
2022-11-23T01:19:11.8730174Z Install  1 Package (+4 Dependent packages)
2022-11-23T01:19:11.8730380Z 
2022-11-23T01:19:11.8730513Z Total download size: 4.3 M
2022-11-23T01:19:11.8730757Z Installed size: 12 M
2022-11-23T01:19:11.8731020Z Downloading packages:
2022-11-23T01:19:11.9833948Z --------------------------------------------------------------------------------
2022-11-23T01:19:11.9834642Z Total                                               39 MB/s | 4.3 MB  00:00     
2022-11-23T01:19:11.9880341Z Running transaction check
2022-11-23T01:19:12.0051352Z Running transaction test
2022-11-23T01:19:12.0216797Z Transaction test succeeded
2022-11-23T01:19:12.0218305Z Running transaction
2022-11-23T01:19:12.5458090Z   Installing : nvidia-container-toolkit-base-1.11.0-1.x86_64                1/5 
2022-11-23T01:19:12.6218323Z   Installing : libnvidia-container1-1.11.0-1.x86_64                         2/5 
2022-11-23T01:19:12.7401275Z   Installing : libnvidia-container-tools-1.11.0-1.x86_64                    3/5 
2022-11-23T01:19:12.7843222Z   Installing : nvidia-container-toolkit-1.11.0-1.x86_64                     4/5 
2022-11-23T01:19:12.8394495Z   Installing : nvidia-docker2-2.11.0-1.noarch                               5/5 
2022-11-23T01:19:12.8539848Z   Verifying  : libnvidia-container1-1.11.0-1.x86_64                         1/5 
2022-11-23T01:19:12.8686042Z   Verifying  : nvidia-container-toolkit-base-1.11.0-1.x86_64                2/5 
2022-11-23T01:19:12.8793629Z   Verifying  : nvidia-container-toolkit-1.11.0-1.x86_64                     3/5 
2022-11-23T01:19:12.8949973Z   Verifying  : libnvidia-container-tools-1.11.0-1.x86_64                    4/5 
2022-11-23T01:19:12.9746024Z   Verifying  : nvidia-docker2-2.11.0-1.noarch                               5/5 
2022-11-23T01:19:12.9746312Z 
2022-11-23T01:19:12.9746420Z Installed:
2022-11-23T01:19:12.9747258Z   nvidia-docker2.noarch 0:2.11.0-1                                              
2022-11-23T01:19:12.9747506Z 
2022-11-23T01:19:12.9747615Z Dependency Installed:
2022-11-23T01:19:12.9748105Z   libnvidia-container-tools.x86_64 0:1.11.0-1                                   
2022-11-23T01:19:12.9748581Z   libnvidia-container1.x86_64 0:1.11.0-1                                        
2022-11-23T01:19:12.9749047Z   nvidia-container-toolkit.x86_64 0:1.11.0-1                                    
2022-11-23T01:19:12.9749512Z   nvidia-container-toolkit-base.x86_64 0:1.11.0-1                               
2022-11-23T01:19:12.9749745Z 
2022-11-23T01:19:12.9749847Z Complete!
2022-11-23T01:19:13.0780370Z + sudo systemctl restart docker
2022-11-23T01:19:21.2765830Z Command completed after 1 attempt(s).
2022-11-23T01:19:21.2766072Z 
2022-11-23T01:19:21.2769752Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:19:21.2818376Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-11-23T01:19:21.2818777Z python3 -m pip install psutil==5.9.1
2022-11-23T01:19:21.2819086Z python3 -m pip install pynvml==11.4.1
2022-11-23T01:19:21.2819442Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-11-23T01:19:21.2819829Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2022-11-23T01:19:21.2833334Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:19:21.2833634Z env:
2022-11-23T01:19:21.2833876Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:21.2834127Z   GPU_FLAG: --gpus all
2022-11-23T01:19:21.2834380Z ##[endgroup]
2022-11-23T01:19:22.5221186Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:19:22.8997289Z Collecting psutil==5.9.1
2022-11-23T01:19:22.9176876Z   Downloading psutil-5.9.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
2022-11-23T01:19:22.9913954Z Installing collected packages: psutil
2022-11-23T01:19:23.1504884Z Successfully installed psutil-5.9.1
2022-11-23T01:19:23.6378683Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T01:19:23.7216169Z Collecting pynvml==11.4.1
2022-11-23T01:19:23.7393142Z   Downloading pynvml-11.4.1-py3-none-any.whl (46 kB)
2022-11-23T01:19:23.7891114Z Installing collected packages: pynvml
2022-11-23T01:19:23.8472415Z Successfully installed pynvml-11.4.1
2022-11-23T01:19:23.8984375Z Prepare all required actions
2022-11-23T01:19:23.8984744Z Getting action download info
2022-11-23T01:19:24.0636054Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2022-11-23T01:19:24.2530308Z Download action repository 'actions/download-artifact@v3' (SHA:9782bd6a9848b53b110e712e20e42d89988822b7)
2022-11-23T01:19:24.3869133Z ##[group]Run ./.github/actions/download-build-artifacts
2022-11-23T01:19:24.3869420Z with:
2022-11-23T01:19:24.3869693Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T01:19:24.3869958Z env:
2022-11-23T01:19:24.3870175Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:24.3870425Z   GPU_FLAG: --gpus all
2022-11-23T01:19:24.3870659Z ##[endgroup]
2022-11-23T01:19:24.3900519Z ##[group]Run seemethere/download-artifact-s3@v4
2022-11-23T01:19:24.3900813Z with:
2022-11-23T01:19:24.3901073Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T01:19:24.3901384Z   s3-bucket: gha-artifacts
2022-11-23T01:19:24.3901687Z   region: us-east-1
2022-11-23T01:19:24.3901923Z env:
2022-11-23T01:19:24.3902156Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:24.3902401Z   GPU_FLAG: --gpus all
2022-11-23T01:19:24.3902638Z ##[endgroup]
2022-11-23T01:19:24.9272004Z Found 1 objects with prefix pytorch/pytorch/3528293562/linux-bionic-cuda11.6-py3.10-gcc7/
2022-11-23T01:19:24.9273192Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-11-23T01:19:37.2490381Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-11-23T01:19:37.2490743Z 
2022-11-23T01:19:37.2495279Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-11-23T01:19:37.2496580Z Artifact download has finished successfully
2022-11-23T01:19:37.2689956Z ##[group]Run unzip -o artifacts.zip
2022-11-23T01:19:37.2690277Z unzip -o artifacts.zip
2022-11-23T01:19:37.2703804Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:19:37.2704107Z env:
2022-11-23T01:19:37.2704347Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:37.2704594Z   GPU_FLAG: --gpus all
2022-11-23T01:19:37.2704844Z ##[endgroup]
2022-11-23T01:19:37.2818189Z Archive:  artifacts.zip
2022-11-23T01:19:37.2819738Z    creating: dist/
2022-11-23T01:19:39.3487503Z   inflating: dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl  
2022-11-23T01:19:39.3488428Z    creating: build/custom_test_artifacts/
2022-11-23T01:19:39.3489015Z    creating: build/custom_test_artifacts/custom-op-build/
2022-11-23T01:19:39.3489491Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-11-23T01:19:39.3495851Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:19:39.3497074Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-11-23T01:19:39.3497968Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:19:39.3498522Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:19:39.3499093Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:19:39.3500629Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:19:39.3502098Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:19:39.3503390Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:19:39.3503940Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:19:39.3505526Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:19:39.3507193Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:19:39.3508731Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:19:39.3509875Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:19:39.3511208Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:19:39.3512616Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:19:39.3513397Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:19:39.3513965Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:19:39.3567633Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:19:39.3569229Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:19:39.3570886Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:19:39.3572553Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:19:39.3574076Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:19:39.3574800Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:19:39.3575494Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:19:39.3576197Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:19:39.3576876Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:19:39.3616234Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:19:39.3657491Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:19:39.3659013Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:19:39.3660530Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:19:39.3661912Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:19:39.3663225Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:19:39.3664189Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:19:39.3665398Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:19:39.3666029Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:19:39.3737652Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:19:39.3810782Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:19:39.3812224Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:19:39.3813417Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-11-23T01:19:39.3814784Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-11-23T01:19:39.3816020Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:19:39.3817235Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-11-23T01:19:39.3818430Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-11-23T01:19:39.3819812Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-11-23T01:19:39.3820943Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-11-23T01:19:39.3821522Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-11-23T01:19:39.3822097Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-11-23T01:19:39.3822723Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-11-23T01:19:39.3823322Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-11-23T01:19:39.3823886Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-11-23T01:19:39.3824473Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-11-23T01:19:39.3841742Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-11-23T01:19:39.3955890Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-11-23T01:19:39.3957146Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-11-23T01:19:39.3958493Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-11-23T01:19:39.3959885Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-11-23T01:19:39.3961266Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-11-23T01:19:39.3962589Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-11-23T01:19:39.3963380Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-11-23T01:19:39.3963981Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-11-23T01:19:39.3964591Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-11-23T01:19:39.3965184Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-11-23T01:19:39.3965778Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-11-23T01:19:39.3983813Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-11-23T01:19:39.4066584Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-11-23T01:19:39.4068258Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:19:39.4069554Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:19:39.4070771Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-11-23T01:19:39.4071911Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-11-23T01:19:39.4072424Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:19:39.4072952Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-11-23T01:19:39.4074456Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-11-23T01:19:39.4075615Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-11-23T01:19:39.4076478Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-11-23T01:19:39.4168681Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-11-23T01:19:39.4231564Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-11-23T01:19:39.4232571Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-11-23T01:19:39.4233027Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-11-23T01:19:39.4239290Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:19:39.4240502Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-11-23T01:19:39.4241277Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:19:39.4241835Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:19:39.4242387Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:19:39.4243659Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:19:39.4244899Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:19:39.4246044Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:19:39.4246622Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:19:39.4248345Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:19:39.4249820Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:19:39.4251381Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:19:39.4252707Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:19:39.4254291Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:19:39.4255690Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:19:39.4256519Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:19:39.4257089Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:19:39.4311476Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:19:39.4313039Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:19:39.4314670Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:19:39.4316322Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:19:39.4317840Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:19:39.4318547Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:19:39.4319355Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:19:39.4320044Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:19:39.4320835Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:19:39.4360572Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:19:39.4401947Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:19:39.4403477Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:19:39.4404951Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:19:39.4406347Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:19:39.4407638Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:19:39.4408579Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:19:39.4409801Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:19:39.4410446Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:19:39.4482229Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:19:39.4555602Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:19:39.4556998Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:19:39.4558200Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-11-23T01:19:39.4559323Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-11-23T01:19:39.4560559Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:19:39.4561760Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-11-23T01:19:39.4563093Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-11-23T01:19:39.4564459Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-11-23T01:19:39.4565273Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-11-23T01:19:39.4565847Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-11-23T01:19:39.4566447Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-11-23T01:19:39.4567011Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-11-23T01:19:39.4567609Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-11-23T01:19:39.4568198Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-11-23T01:19:39.4568771Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-11-23T01:19:39.4585768Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-11-23T01:19:39.4649265Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-11-23T01:19:39.4650890Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:19:39.4652179Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:19:39.4653385Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-11-23T01:19:39.4654430Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-11-23T01:19:39.4654976Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:19:39.4655482Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-11-23T01:19:39.4656782Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-11-23T01:19:39.4657860Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-11-23T01:19:39.4658717Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-11-23T01:19:39.4707127Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-11-23T01:19:39.4708460Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-11-23T01:19:39.4708952Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-11-23T01:19:39.4715470Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-11-23T01:19:39.4716753Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-11-23T01:19:39.4717499Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-11-23T01:19:39.4718094Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-11-23T01:19:39.4718677Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-11-23T01:19:39.4720025Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-11-23T01:19:39.4721377Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-11-23T01:19:39.4722714Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-11-23T01:19:39.4723384Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-11-23T01:19:39.4725328Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-11-23T01:19:39.4726815Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-11-23T01:19:39.4728371Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-11-23T01:19:39.4729557Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-11-23T01:19:39.4730963Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-11-23T01:19:39.4732432Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-11-23T01:19:39.4733319Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-11-23T01:19:39.4733915Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-11-23T01:19:39.4787659Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-11-23T01:19:39.4789370Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-11-23T01:19:39.4791003Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-11-23T01:19:39.4792734Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-11-23T01:19:39.4794284Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-11-23T01:19:39.4795008Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-11-23T01:19:39.4795792Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-11-23T01:19:39.4796530Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-11-23T01:19:39.4797259Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-11-23T01:19:39.4836097Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-11-23T01:19:39.4877251Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-11-23T01:19:39.4878874Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-11-23T01:19:39.4880442Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-11-23T01:19:39.4881912Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-11-23T01:19:39.4883054Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-11-23T01:19:39.4884330Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-11-23T01:19:39.4885198Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-11-23T01:19:39.4885870Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-11-23T01:19:39.4957708Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-11-23T01:19:39.5030808Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-11-23T01:19:39.5032315Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-11-23T01:19:39.5033574Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-11-23T01:19:39.5034831Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-11-23T01:19:39.5036059Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-11-23T01:19:39.5037374Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-11-23T01:19:39.5038666Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-11-23T01:19:39.5040140Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-11-23T01:19:39.5041192Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-11-23T01:19:39.5041817Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-11-23T01:19:39.5042442Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-11-23T01:19:39.5043049Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-11-23T01:19:39.5043678Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-11-23T01:19:39.5044474Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-11-23T01:19:39.5045883Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-11-23T01:19:39.5046527Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-11-23T01:19:39.5193016Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-11-23T01:19:39.5194484Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-11-23T01:19:39.5195888Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-11-23T01:19:39.5197314Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-11-23T01:19:39.5198794Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-11-23T01:19:39.5199959Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-11-23T01:19:39.5200608Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-11-23T01:19:39.5201231Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-11-23T01:19:39.5201874Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-11-23T01:19:39.5202512Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-11-23T01:19:39.5203153Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-11-23T01:19:39.5220699Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-11-23T01:19:39.5279332Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-11-23T01:19:39.5280874Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-11-23T01:19:39.5282246Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-11-23T01:19:39.5283564Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-11-23T01:19:39.5284438Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-11-23T01:19:39.5284997Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-11-23T01:19:39.5285548Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-11-23T01:19:39.5286650Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-11-23T01:19:39.5287776Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-11-23T01:19:39.5288860Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-11-23T01:19:39.5407716Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-11-23T01:19:39.5453059Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-11-23T01:19:39.5453840Z    creating: build/lib/
2022-11-23T01:19:39.5518861Z   inflating: build/lib/libgtest.a    
2022-11-23T01:19:39.5519759Z   inflating: build/lib/libclog.a     
2022-11-23T01:19:39.5530452Z   inflating: build/lib/libpthreadpool.a  
2022-11-23T01:19:39.5636728Z   inflating: build/lib/libprotobuf-lite.a  
2022-11-23T01:19:39.5645793Z   inflating: build/lib/libittnotify.a  
2022-11-23T01:19:39.5737366Z   inflating: build/lib/libbenchmark.a  
2022-11-23T01:19:39.5769239Z   inflating: build/lib/libtensorpipe_uv.a  
2022-11-23T01:19:39.5846183Z   inflating: build/lib/libasmjit.a   
2022-11-23T01:19:39.6379406Z   inflating: build/lib/libprotobuf.a  
2022-11-23T01:19:39.6512292Z   inflating: build/lib/libgloo.a     
2022-11-23T01:19:39.6544496Z   inflating: build/lib/libfmt.a      
2022-11-23T01:19:39.6546003Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-11-23T01:19:39.6547096Z   inflating: build/lib/libfoxi_loader.a  
2022-11-23T01:19:39.6626565Z   inflating: build/lib/libc10.so     
2022-11-23T01:19:39.6628055Z   inflating: build/lib/libtorch_global_deps.so  
2022-11-23T01:19:39.6638001Z   inflating: build/lib/libcpuinfo.a  
2022-11-23T01:19:39.6646932Z   inflating: build/lib/libcpuinfo_internals.a  
2022-11-23T01:19:39.7216907Z   inflating: build/lib/libprotoc.a   
2022-11-23T01:19:39.7233996Z   inflating: build/lib/libqnnpack.a  
2022-11-23T01:19:39.7236623Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-11-23T01:19:39.7259382Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-11-23T01:19:39.7278496Z   inflating: build/lib/libgmock.a    
2022-11-23T01:19:39.7300571Z   inflating: build/lib/libnnpack.a   
2022-11-23T01:19:39.7301429Z   inflating: build/lib/libgtest_main.a  
2022-11-23T01:19:39.7302245Z   inflating: build/lib/libbenchmark_main.a  
2022-11-23T01:19:40.7087767Z   inflating: build/lib/libdnnl.a     
2022-11-23T01:19:40.7229333Z   inflating: build/lib/libXNNPACK.a  
2022-11-23T01:19:40.7884624Z   inflating: build/lib/libtensorpipe.a  
2022-11-23T01:19:40.7937826Z   inflating: build/lib/libc10_cuda.so  
2022-11-23T01:19:40.9480480Z   inflating: build/lib/libfbgemm.a   
2022-11-23T01:19:40.9481116Z   inflating: build/lib/libgmock_main.a  
2022-11-23T01:19:41.0627084Z   inflating: build/lib/libdnnl_graph.a  
2022-11-23T01:19:41.1143074Z   inflating: build/lib/libkineto.a   
2022-11-23T01:19:41.1433258Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-11-23T01:19:41.1478622Z   inflating: build/lib/libcaffe2_protos.a  
2022-11-23T01:19:41.1526758Z   inflating: build/lib/libonnx_proto.a  
2022-11-23T01:19:41.2205329Z   inflating: build/lib/libonnx.a     
2022-11-23T01:19:41.2639503Z   inflating: build/lib/libgloo_cuda.a  
2022-11-23T01:19:43.6276780Z   inflating: build/lib/libtorch_cpu.so  
2022-11-23T01:19:45.7662537Z   inflating: build/lib/libtorch_cuda.so  
2022-11-23T01:19:45.7663702Z   inflating: build/lib/libtorch.so   
2022-11-23T01:19:46.7552619Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-11-23T01:19:46.7555108Z   inflating: build/lib/libc10d_cuda_test.so  
2022-11-23T01:19:46.7578937Z   inflating: build/lib/libjitbackend_test.so  
2022-11-23T01:19:46.7639238Z   inflating: build/lib/libtorchbind_test.so  
2022-11-23T01:19:46.7670146Z   inflating: build/lib/libbackend_with_compiler.so  
2022-11-23T01:19:46.7674932Z   inflating: build/lib/libshm.so     
2022-11-23T01:19:46.9479200Z   inflating: build/lib/libtorch_python.so  
2022-11-23T01:19:46.9518810Z   inflating: build/lib/libnnapi_backend.so  
2022-11-23T01:19:46.9519452Z    creating: build/bin/
2022-11-23T01:19:46.9571693Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-11-23T01:19:46.9626605Z   inflating: build/bin/c10_DeviceGuard_test  
2022-11-23T01:19:46.9680500Z   inflating: build/bin/c10_Device_test  
2022-11-23T01:19:46.9743314Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-11-23T01:19:46.9794770Z   inflating: build/bin/c10_StreamGuard_test  
2022-11-23T01:19:46.9847022Z   inflating: build/bin/c10_SymInt_test  
2022-11-23T01:19:46.9906761Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-11-23T01:19:46.9966238Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-11-23T01:19:47.0027160Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-11-23T01:19:47.0078680Z   inflating: build/bin/c10_Array_test  
2022-11-23T01:19:47.0135106Z   inflating: build/bin/c10_Bitset_test  
2022-11-23T01:19:47.0190041Z   inflating: build/bin/c10_C++17_test  
2022-11-23T01:19:47.0241404Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-11-23T01:19:47.0293875Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-11-23T01:19:47.0346733Z   inflating: build/bin/c10_Half_test  
2022-11-23T01:19:47.0413777Z   inflating: build/bin/c10_Metaprogramming_test  
2022-11-23T01:19:47.0474655Z   inflating: build/bin/c10_LeftRight_test  
2022-11-23T01:19:47.0630465Z   inflating: build/bin/c10_SmallVectorTest  
2022-11-23T01:19:47.0684113Z   inflating: build/bin/c10_Synchronized_test  
2022-11-23T01:19:47.0745025Z   inflating: build/bin/c10_ThreadLocal_test  
2022-11-23T01:19:47.0801841Z   inflating: build/bin/c10_TypeIndex_test  
2022-11-23T01:19:47.0855392Z   inflating: build/bin/c10_TypeList_test  
2022-11-23T01:19:47.0906723Z   inflating: build/bin/c10_TypeTraits_test  
2022-11-23T01:19:47.0962143Z   inflating: build/bin/c10_accumulate_test  
2022-11-23T01:19:47.1021751Z   inflating: build/bin/c10_bfloat16_test  
2022-11-23T01:19:47.1079405Z   inflating: build/bin/c10_complex_math_test  
2022-11-23T01:19:47.1138584Z   inflating: build/bin/c10_complex_test  
2022-11-23T01:19:47.1256538Z   inflating: build/bin/c10_either_test  
2022-11-23T01:19:47.1312607Z   inflating: build/bin/c10_exception_test  
2022-11-23T01:19:47.1365911Z   inflating: build/bin/c10_flags_test  
2022-11-23T01:19:47.1549062Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-11-23T01:19:47.1603038Z   inflating: build/bin/c10_irange_test  
2022-11-23T01:19:47.1664681Z   inflating: build/bin/c10_logging_test  
2022-11-23T01:19:47.1744628Z   inflating: build/bin/c10_optional_test  
2022-11-23T01:19:47.1810988Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-11-23T01:19:47.1869375Z   inflating: build/bin/c10_registry_test  
2022-11-23T01:19:47.1932637Z   inflating: build/bin/c10_string_view_test  
2022-11-23T01:19:47.1987349Z   inflating: build/bin/c10_tempfile_test  
2022-11-23T01:19:47.2048010Z   inflating: build/bin/c10_typeid_test  
2022-11-23T01:19:47.2108589Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-11-23T01:19:47.2630509Z   inflating: build/bin/protoc-3.13.0.0  
2022-11-23T01:19:47.3152314Z   inflating: build/bin/protoc        
2022-11-23T01:19:47.3204017Z   inflating: build/bin/c10_cuda_CUDATest  
2022-11-23T01:19:47.3520348Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-11-23T01:19:47.3875981Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-11-23T01:19:47.3933329Z   inflating: build/bin/FileStoreTest  
2022-11-23T01:19:47.3991060Z   inflating: build/bin/HashStoreTest  
2022-11-23T01:19:47.4055472Z   inflating: build/bin/TCPStoreTest  
2022-11-23T01:19:47.4071519Z   inflating: build/bin/ProcessGroupMPITest  
2022-11-23T01:19:47.4074750Z   inflating: build/bin/example_allreduce  
2022-11-23T01:19:47.4130968Z   inflating: build/bin/Dimname_test  
2022-11-23T01:19:47.4199392Z   inflating: build/bin/MaybeOwned_test  
2022-11-23T01:19:47.4277881Z   inflating: build/bin/Dict_test     
2022-11-23T01:19:47.4338926Z   inflating: build/bin/NamedTensor_test  
2022-11-23T01:19:47.4402348Z   inflating: build/bin/apply_utils_test  
2022-11-23T01:19:47.4465323Z   inflating: build/bin/atest         
2022-11-23T01:19:47.4522942Z   inflating: build/bin/broadcast_test  
2022-11-23T01:19:47.4585346Z   inflating: build/bin/cpu_generator_test  
2022-11-23T01:19:47.4651418Z   inflating: build/bin/basic         
2022-11-23T01:19:47.4707292Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-11-23T01:19:47.4760642Z   inflating: build/bin/dispatch_key_set_test  
2022-11-23T01:19:47.4813559Z   inflating: build/bin/dlconvertor_test  
2022-11-23T01:19:47.4908882Z   inflating: build/bin/cpu_rng_test  
2022-11-23T01:19:47.4971808Z   inflating: build/bin/extension_backend_test  
2022-11-23T01:19:47.5031722Z   inflating: build/bin/half_test     
2022-11-23T01:19:47.5133341Z   inflating: build/bin/ivalue_test   
2022-11-23T01:19:47.5185979Z   inflating: build/bin/lazy_tensor_test  
2022-11-23T01:19:47.5243706Z   inflating: build/bin/math_kernel_test  
2022-11-23T01:19:47.5300808Z   inflating: build/bin/memory_overlapping_test  
2022-11-23T01:19:47.5358376Z   inflating: build/bin/memory_format_test  
2022-11-23T01:19:47.5413990Z   inflating: build/bin/mobile_memory_cleanup  
2022-11-23T01:19:47.5468501Z   inflating: build/bin/operator_name_test  
2022-11-23T01:19:47.5528522Z   inflating: build/bin/native_test   
2022-11-23T01:19:47.5582077Z   inflating: build/bin/operators_test  
2022-11-23T01:19:47.5638418Z   inflating: build/bin/packedtensoraccessor_test  
2022-11-23T01:19:47.5708592Z   inflating: build/bin/pow_test      
2022-11-23T01:19:47.5769727Z   inflating: build/bin/quantized_test  
2022-11-23T01:19:47.5822630Z   inflating: build/bin/reduce_ops_test  
2022-11-23T01:19:47.5876938Z   inflating: build/bin/reportMemoryUsage_test  
2022-11-23T01:19:47.5937473Z   inflating: build/bin/scalar_tensor_test  
2022-11-23T01:19:47.5998853Z   inflating: build/bin/scalar_test   
2022-11-23T01:19:47.6054501Z   inflating: build/bin/stride_properties_test  
2022-11-23T01:19:47.6139203Z   inflating: build/bin/tensor_iterator_test  
2022-11-23T01:19:47.6199085Z   inflating: build/bin/test_parallel  
2022-11-23T01:19:47.6259099Z   inflating: build/bin/type_ptr_test  
2022-11-23T01:19:47.6261914Z   inflating: build/bin/thread_init_test  
2022-11-23T01:19:47.6314771Z   inflating: build/bin/variant_test  
2022-11-23T01:19:47.6379554Z   inflating: build/bin/type_test     
2022-11-23T01:19:47.6435479Z   inflating: build/bin/undefined_tensor_test  
2022-11-23T01:19:47.6489829Z   inflating: build/bin/weakref_test  
2022-11-23T01:19:47.6490971Z   inflating: build/bin/verify_api_visibility  
2022-11-23T01:19:47.6566020Z   inflating: build/bin/vmap_test     
2022-11-23T01:19:47.6620303Z   inflating: build/bin/wrapdim_test  
2022-11-23T01:19:47.6684457Z   inflating: build/bin/IListRef_test  
2022-11-23T01:19:47.6736542Z   inflating: build/bin/xla_tensor_test  
2022-11-23T01:19:47.6855071Z   inflating: build/bin/List_test     
2022-11-23T01:19:47.6985995Z   inflating: build/bin/kernel_function_legacy_test  
2022-11-23T01:19:47.7089826Z   inflating: build/bin/kernel_function_test  
2022-11-23T01:19:47.7159723Z   inflating: build/bin/KernelFunction_test  
2022-11-23T01:19:47.7298205Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-11-23T01:19:47.7409783Z   inflating: build/bin/kernel_lambda_test  
2022-11-23T01:19:47.7473985Z   inflating: build/bin/kernel_stackbased_test  
2022-11-23T01:19:47.7527841Z   inflating: build/bin/CppSignature_test  
2022-11-23T01:19:47.7631155Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-11-23T01:19:47.7682347Z   inflating: build/bin/op_allowlist_test  
2022-11-23T01:19:47.7742621Z   inflating: build/bin/backend_fallback_test  
2022-11-23T01:19:47.8056639Z   inflating: build/bin/op_registration_test  
2022-11-23T01:19:47.8113102Z   inflating: build/bin/inline_container_test  
2022-11-23T01:19:47.8168776Z   inflating: build/bin/cuda_apply_test  
2022-11-23T01:19:47.8233277Z   inflating: build/bin/cuda_atomic_ops_test  
2022-11-23T01:19:47.8290581Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-11-23T01:19:47.8362836Z   inflating: build/bin/cuda_complex_math_test  
2022-11-23T01:19:47.8415206Z   inflating: build/bin/cuda_device_test  
2022-11-23T01:19:47.8477786Z   inflating: build/bin/cuda_complex_test  
2022-11-23T01:19:47.8541025Z   inflating: build/bin/cuda_cub_test  
2022-11-23T01:19:47.8594575Z   inflating: build/bin/cuda_dlconvertor_test  
2022-11-23T01:19:47.8648385Z   inflating: build/bin/cuda_integer_divider_test  
2022-11-23T01:19:47.8720482Z   inflating: build/bin/cuda_distributions_test  
2022-11-23T01:19:47.8783197Z   inflating: build/bin/cuda_generator_test  
2022-11-23T01:19:47.8835999Z   inflating: build/bin/cuda_half_test  
2022-11-23T01:19:47.8900742Z   inflating: build/bin/cuda_stream_test  
2022-11-23T01:19:47.8952846Z   inflating: build/bin/cuda_optional_test  
2022-11-23T01:19:47.9008822Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-11-23T01:19:47.9063573Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-11-23T01:19:47.9115727Z   inflating: build/bin/cuda_cudnn_test  
2022-11-23T01:19:47.9171227Z   inflating: build/bin/cuda_vectorized_test  
2022-11-23T01:19:47.9189096Z   inflating: build/bin/tutorial_tensorexpr  
2022-11-23T01:19:47.9259470Z   inflating: build/bin/ProcessGroupGlooTest  
2022-11-23T01:19:47.9322092Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-11-23T01:19:47.9387832Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-11-23T01:19:47.9450629Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-11-23T01:19:47.9507035Z   inflating: build/bin/ProcessGroupUCCTest  
2022-11-23T01:19:47.9565023Z   inflating: build/bin/test_dist_autograd  
2022-11-23T01:19:47.9567525Z   inflating: build/bin/parallel_benchmark  
2022-11-23T01:19:47.9641541Z   inflating: build/bin/test_mobile_nnc  
2022-11-23T01:19:47.9716715Z   inflating: build/bin/test_cpp_rpc  
2022-11-23T01:19:47.9727977Z   inflating: build/bin/aot_model_compiler_test  
2022-11-23T01:19:48.0642809Z   inflating: build/bin/test_tensorexpr  
2022-11-23T01:19:48.0648438Z   inflating: build/bin/torch_shm_manager  
2022-11-23T01:19:48.1035220Z   inflating: build/bin/test_lazy     
2022-11-23T01:19:48.2360747Z   inflating: build/bin/test_api      
2022-11-23T01:19:48.3570457Z   inflating: build/bin/test_jit      
2022-11-23T01:19:48.3572151Z   inflating: .pytorch-test-times.json  
2022-11-23T01:19:48.3601700Z ##[group]Run df -H
2022-11-23T01:19:48.3601959Z df -H
2022-11-23T01:19:48.3615142Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T01:19:48.3615440Z env:
2022-11-23T01:19:48.3615679Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:48.3615935Z   GPU_FLAG: --gpus all
2022-11-23T01:19:48.3616186Z ##[endgroup]
2022-11-23T01:19:48.3656419Z Filesystem      Size  Used Avail Use% Mounted on
2022-11-23T01:19:48.3657004Z devtmpfs        129G     0  129G   0% /dev
2022-11-23T01:19:48.3657288Z tmpfs           129G     0  129G   0% /dev/shm
2022-11-23T01:19:48.3657955Z tmpfs           129G  607k  129G   1% /run
2022-11-23T01:19:48.3658558Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2022-11-23T01:19:48.3659163Z /dev/xvda1      162G   30G  132G  19% /
2022-11-23T01:19:48.3659579Z tmpfs            26G     0   26G   0% /run/user/0
2022-11-23T01:19:48.3681311Z ##[group]Run .github/scripts/parse_ref.py
2022-11-23T01:19:48.3681650Z .github/scripts/parse_ref.py
2022-11-23T01:19:48.3693142Z shell: /usr/bin/bash -e {0}
2022-11-23T01:19:48.3693396Z env:
2022-11-23T01:19:48.3693616Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:48.3693886Z   GPU_FLAG: --gpus all
2022-11-23T01:19:48.3694135Z ##[endgroup]
2022-11-23T01:19:48.3992782Z ##[group]Run set -x
2022-11-23T01:19:48.3993165Z set -x
2022-11-23T01:19:48.3993394Z 
2022-11-23T01:19:48.3993648Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-11-23T01:19:48.3994000Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-11-23T01:19:48.3994350Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-11-23T01:19:48.3994658Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-11-23T01:19:48.3994930Z else
2022-11-23T01:19:48.3995205Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-11-23T01:19:48.3995491Z fi
2022-11-23T01:19:48.3995691Z 
2022-11-23T01:19:48.3996010Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-11-23T01:19:48.3996329Z 
2022-11-23T01:19:48.3996608Z # sanitize the input commit message and PR body here:
2022-11-23T01:19:48.3996901Z #
2022-11-23T01:19:48.3997283Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-11-23T01:19:48.3997777Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-11-23T01:19:48.3998197Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-11-23T01:19:48.3998513Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-11-23T01:19:48.3998751Z 
2022-11-23T01:19:48.3999101Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-11-23T01:19:48.3999478Z # wreak havoc internally
2022-11-23T01:19:48.3999931Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-11-23T01:19:48.4000243Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-11-23T01:19:48.4000503Z 
2022-11-23T01:19:48.4000811Z # detached container should get cleaned up by teardown_ec2_linux
2022-11-23T01:19:48.4001197Z # TODO: Stop building test binaries as part of the build phase
2022-11-23T01:19:48.4001569Z # Used for GPU_FLAG since that doesn't play nice
2022-11-23T01:19:48.4001898Z # shellcheck disable=SC2086,SC2090
2022-11-23T01:19:48.4002199Z container_name=$(docker run \
2022-11-23T01:19:48.4002461Z   ${GPU_FLAG:-} \
2022-11-23T01:19:48.4002731Z   -e BUILD_ENVIRONMENT \
2022-11-23T01:19:48.4003001Z   -e PR_NUMBER \
2022-11-23T01:19:48.4003249Z   -e GITHUB_ACTIONS \
2022-11-23T01:19:48.4003510Z   -e BASE_SHA \
2022-11-23T01:19:48.4003759Z   -e BRANCH \
2022-11-23T01:19:48.4003990Z   -e SHA1 \
2022-11-23T01:19:48.4004249Z   -e AWS_DEFAULT_REGION \
2022-11-23T01:19:48.4004520Z   -e IN_WHEEL_TEST \
2022-11-23T01:19:48.4004764Z   -e SHARD_NUMBER \
2022-11-23T01:19:48.4005022Z   -e TEST_CONFIG \
2022-11-23T01:19:48.4005285Z   -e NUM_TEST_SHARDS \
2022-11-23T01:19:48.4005526Z   -e PR_BODY \
2022-11-23T01:19:48.4005788Z   -e COMMIT_MESSAGES \
2022-11-23T01:19:48.4006076Z   -e PYTORCH_RETRY_TEST_CASES \
2022-11-23T01:19:48.4006370Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-11-23T01:19:48.4006656Z   -e PR_LABELS \
2022-11-23T01:19:48.4006944Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-11-23T01:19:48.4007233Z   -e SCCACHE_BUCKET \
2022-11-23T01:19:48.4007490Z   -e SCCACHE_S3_KEY_PREFIX \
2022-11-23T01:19:48.4007759Z   -e XLA_CUDA \
2022-11-23T01:19:48.4008042Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-11-23T01:19:48.4008350Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2022-11-23T01:19:48.4008675Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2022-11-23T01:19:48.4009022Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-11-23T01:19:48.4009328Z   --ulimit stack=10485760:83886080 \
2022-11-23T01:19:48.4009640Z   --security-opt seccomp=unconfined \
2022-11-23T01:19:48.4010018Z   --cap-add=SYS_PTRACE \
2022-11-23T01:19:48.4010298Z   --ipc=host \
2022-11-23T01:19:48.4010546Z   --shm-size="${SHM_SIZE}" \
2022-11-23T01:19:48.4010806Z   --tty \
2022-11-23T01:19:48.4011043Z   --detach \
2022-11-23T01:19:48.4011295Z   --name="${container_name}" \
2022-11-23T01:19:48.4011570Z   --user jenkins \
2022-11-23T01:19:48.4011889Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-11-23T01:19:48.4012212Z   -w /var/lib/jenkins/workspace \
2022-11-23T01:19:48.4012492Z   "${DOCKER_IMAGE}"
2022-11-23T01:19:48.4012737Z )
2022-11-23T01:19:48.4013019Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2022-11-23T01:19:48.4013459Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2022-11-23T01:19:48.4025776Z shell: /usr/bin/bash -e {0}
2022-11-23T01:19:48.4026009Z env:
2022-11-23T01:19:48.4026255Z   GIT_DEFAULT_BRANCH: master
2022-11-23T01:19:48.4026522Z   GPU_FLAG: --gpus all
2022-11-23T01:19:48.4027213Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T01:19:48.4027598Z   PR_NUMBER: 
2022-11-23T01:19:48.4027835Z   BRANCH: master
2022-11-23T01:19:48.4028106Z   SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:19:48.4028407Z   BASE_SHA: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:19:48.4028702Z   PYTORCH_RETRY_TEST_CASES: 1
2022-11-23T01:19:48.4028986Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-11-23T01:19:48.4029247Z   TEST_CONFIG: distributed
2022-11-23T01:19:48.4029628Z   SHARD_NUMBER: 1
2022-11-23T01:19:48.4029868Z   NUM_TEST_SHARDS: 3
2022-11-23T01:19:48.4030087Z   PR_BODY: 
2022-11-23T01:19:48.4030388Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-11-23T01:19:48.4030717Z   SCCACHE_S3_KEY_PREFIX: pull
2022-11-23T01:19:48.4030952Z   SHM_SIZE: 2g
2022-11-23T01:19:48.4031443Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:19:48.4031913Z   XLA_CUDA: 
2022-11-23T01:19:48.4032263Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-11-23T01:19:48.4032625Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2022-11-23T01:19:48.4032928Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2022-11-23T01:19:48.4033199Z ##[endgroup]
2022-11-23T01:19:48.4061386Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-11-23T01:19:48.4061900Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *onnx* ]]
2022-11-23T01:19:48.4062239Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-11-23T01:19:48.4065412Z ++ git cherry -v origin/master
2022-11-23T01:19:48.4082731Z + COMMIT_MESSAGES=
2022-11-23T01:19:48.4082997Z + COMMIT_MESSAGES=
2022-11-23T01:19:48.4083220Z + PR_BODY=
2022-11-23T01:19:48.4083470Z + export COMMIT_MESSAGES=
2022-11-23T01:19:48.4083725Z + COMMIT_MESSAGES=
2022-11-23T01:19:48.4083964Z + export PR_BODY=
2022-11-23T01:19:48.4084179Z + PR_BODY=
2022-11-23T01:19:48.4093767Z +++ nproc --ignore=2
2022-11-23T01:19:48.4143644Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3528293562 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T01:20:02.7281689Z + container_name=103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T01:20:02.7282266Z + echo DOCKER_CONTAINER_ID=103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T01:20:02.7287322Z ++ echo dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl
2022-11-23T01:20:02.7289413Z + docker exec -t 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70 sh -c 'pip install dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2022-11-23T01:20:03.2869208Z Processing ./dist/torch-1.14.0a0+git1cfd385-cp310-cp310-linux_x86_64.whl
2022-11-23T01:20:04.2487957Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (1.11.1)
2022-11-23T01:20:04.2492308Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (2.6.3)
2022-11-23T01:20:04.2499251Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (4.4.0)
2022-11-23T01:20:04.2517842Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.10/site-packages (from torch==1.14.0a0+git1cfd385) (3.3.0)
2022-11-23T01:20:04.2598102Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==1.14.0a0+git1cfd385) (1.21.2)
2022-11-23T01:20:04.2814913Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch==1.14.0a0+git1cfd385) (1.2.1)
2022-11-23T01:20:05.1823553Z Installing collected packages: torch
2022-11-23T01:20:15.0850262Z Successfully installed torch-1.14.0a0+git1cfd385
2022-11-23T01:20:15.1508460Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-11-23T01:20:15.1730642Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch
2022-11-23T01:20:15.1731119Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T01:20:15.1731595Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib
2022-11-23T01:20:15.1735273Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test
2022-11-23T01:20:15.1735656Z + BUILD_DIR=build
2022-11-23T01:20:15.1735925Z + BUILD_RENAMED_DIR=build_renamed
2022-11-23T01:20:15.1736196Z + BUILD_BIN_DIR=build/bin
2022-11-23T01:20:15.1736440Z + export VALGRIND=ON
2022-11-23T01:20:15.1736681Z + VALGRIND=ON
2022-11-23T01:20:15.1738850Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *clang9* ]]
2022-11-23T01:20:15.1739349Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *bazel* ]]
2022-11-23T01:20:15.1739715Z ++ realpath build/custom_test_artifacts
2022-11-23T01:20:15.1744848Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-11-23T01:20:15.1748850Z ++ dirname .jenkins/pytorch/test.sh
2022-11-23T01:20:15.1756778Z + source .jenkins/pytorch/common.sh
2022-11-23T01:20:15.1760686Z +++ dirname .jenkins/pytorch/common.sh
2022-11-23T01:20:15.1770924Z ++ source .jenkins/pytorch/common_utils.sh
2022-11-23T01:20:15.1773142Z +++ declare -f -t trap_add
2022-11-23T01:20:15.1779113Z ++ set -ex
2022-11-23T01:20:15.1779502Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-11-23T01:20:15.1779803Z ++ BUILD_TEST_LIBTORCH=0
2022-11-23T01:20:15.1781282Z + echo 'Environment variables'
2022-11-23T01:20:15.1781556Z Environment variables
2022-11-23T01:20:15.1781787Z + env
2022-11-23T01:20:15.1789964Z SHARD_NUMBER=1
2022-11-23T01:20:15.1790287Z NV_LIBCUBLAS_DEV_VERSION=11.9.2.110-1
2022-11-23T01:20:15.1790658Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-6
2022-11-23T01:20:15.1790987Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-11-23T01:20:15.1791437Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.12.10-1+cuda11.6
2022-11-23T01:20:15.1791712Z UCC_HOME=/usr
2022-11-23T01:20:15.1792076Z BUILD_ENVIRONMENT=linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T01:20:15.1792411Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2022-11-23T01:20:15.1792803Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-6=11.6.3.124-1
2022-11-23T01:20:15.1793308Z INSTALLED_DB=yes
2022-11-23T01:20:15.1793579Z HOSTNAME=103428310d4e
2022-11-23T01:20:15.1793828Z GITHUB_REF_NAME=master
2022-11-23T01:20:15.1794123Z GITHUB_API_URL=https://api.github.com
2022-11-23T01:20:15.1794427Z OPENSSL_DIR=/opt/openssl
2022-11-23T01:20:15.1794724Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee
2022-11-23T01:20:15.1795340Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_fffd8e99-fa11-41bd-93f3-a761cb50f3ed
2022-11-23T01:20:15.1795759Z CUDA_PATH=/usr/local/cuda
2022-11-23T01:20:15.1800361Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2022-11-23T01:20:15.1800770Z GITHUB_RUN_ATTEMPT=1
2022-11-23T01:20:15.1801238Z TEST_CONFIG=distributed
2022-11-23T01:20:15.1801840Z NV_LIBNPP_VERSION=11.6.3.124-1
2022-11-23T01:20:15.1802227Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-6=11.6.124-1
2022-11-23T01:20:15.1802552Z GITHUB_REPOSITORY_OWNER=pytorch
2022-11-23T01:20:15.1802830Z GITHUB_ACTIONS=true
2022-11-23T01:20:15.1803075Z NVIDIA_VISIBLE_DEVICES=all
2022-11-23T01:20:15.1803370Z NV_NVPROF_VERSION=11.6.124-1
2022-11-23T01:20:15.1803679Z NV_LIBCUSPARSE_VERSION=11.7.2.124-1
2022-11-23T01:20:15.1803923Z CI=true
2022-11-23T01:20:15.1804171Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2022-11-23T01:20:15.1804566Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-6=11.9.2.110-1
2022-11-23T01:20:15.1804850Z BRANCH=master
2022-11-23T01:20:15.1805086Z GITHUB_HEAD_REF=
2022-11-23T01:20:15.1805459Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2022-11-23T01:20:15.1805770Z GITHUB_ACTOR=pytorchmergebot
2022-11-23T01:20:15.1806307Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2022-11-23T01:20:15.1806609Z GITHUB_ACTION_REF=
2022-11-23T01:20:15.1806889Z NCCL_VERSION=2.12.10-1
2022-11-23T01:20:15.1807134Z GITHUB_ACTION=__self
2022-11-23T01:20:15.1807368Z VALGRIND=ON
2022-11-23T01:20:15.1807621Z GITHUB_REF_PROTECTED=true
2022-11-23T01:20:15.1808069Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2022-11-23T01:20:15.1808462Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2022-11-23T01:20:15.1809041Z ***
2022-11-23T01:20:15.1809257Z INSTALLED_VISION=yes
2022-11-23T01:20:15.1809578Z NVARCH=x86_64
2022-11-23T01:20:15.1809887Z NV_LIBCUSPARSE_DEV_VERSION=11.7.2.124-1
2022-11-23T01:20:15.1810166Z HOME=/var/lib/jenkins
2022-11-23T01:20:15.1810686Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_fffd8e99-fa11-41bd-93f3-a761cb50f3ed
2022-11-23T01:20:15.1811103Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2022-11-23T01:20:15.1811382Z GITHUB_ACTION_REPOSITORY=
2022-11-23T01:20:15.1811651Z GITHUB_REF_TYPE=branch
2022-11-23T01:20:15.1811943Z NV_LIBNCCL_PACKAGE_VERSION=2.12.10-1
2022-11-23T01:20:15.1812227Z GITHUB_RETENTION_DAYS=90
2022-11-23T01:20:15.1812603Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2022-11-23T01:20:15.1812992Z NV_LIBNCCL_PACKAGE=libnccl2=2.12.10-1+cuda11.6
2022-11-23T01:20:15.1813545Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_fffd8e99-fa11-41bd-93f3-a761cb50f3ed
2022-11-23T01:20:15.1813955Z DEBIAN_FRONTEND=noninteractive
2022-11-23T01:20:15.1814289Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2022-11-23T01:20:15.1814581Z GITHUB_REF=refs/heads/master
2022-11-23T01:20:15.1814872Z NV_CUDA_LIB_VERSION=11.6.2-1
2022-11-23T01:20:15.1815156Z GITHUB_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:20:15.1815448Z INSTALLED_PROTOBUF=yes
2022-11-23T01:20:15.1815702Z GITHUB_RUN_ID=3528293562
2022-11-23T01:20:15.1816031Z NV_LIBNPP_PACKAGE=libnpp-11-6=11.6.3.124-1
2022-11-23T01:20:15.1816330Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2022-11-23T01:20:15.1816631Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2022-11-23T01:20:15.1816927Z NV_NVTX_VERSION=11.6.124-1
2022-11-23T01:20:15.1817226Z GITHUB_SERVER_URL=https://github.com
2022-11-23T01:20:15.1817496Z MAX_JOBS=30
2022-11-23T01:20:15.1817757Z NV_LIBCUBLAS_VERSION=11.9.2.110-1
2022-11-23T01:20:15.1818126Z NV_LIBCUBLAS_PACKAGE=libcublas-11-6=11.9.2.110-1
2022-11-23T01:20:15.1818698Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2022-11-23T01:20:15.1819059Z UCX_HOME=/usr
2022-11-23T01:20:15.1819294Z PYTORCH_RETRY_TEST_CASES=1
2022-11-23T01:20:15.1819620Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2022-11-23T01:20:15.1819963Z BASE_SHA=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:20:15.1820296Z NV_CUDA_CUDART_DEV_VERSION=11.6.55-1
2022-11-23T01:20:15.1820573Z PR_BODY=
2022-11-23T01:20:15.1820811Z GITHUB_BASE_REF=
2022-11-23T01:20:15.1821041Z TERM=xterm
2022-11-23T01:20:15.1821268Z XLA_CUDA=
2022-11-23T01:20:15.1821550Z NV_NVML_DEV_VERSION=11.6.55-1
2022-11-23T01:20:15.1821824Z TORCH_CUDA_ARCH_LIST=Maxwell
2022-11-23T01:20:15.1822145Z CUDA_VERSION=11.6.2
2022-11-23T01:20:15.1822511Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-6
2022-11-23T01:20:15.1822812Z OPENSSL_ROOT_DIR=/opt/openssl
2022-11-23T01:20:15.1823388Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_fffd8e99-fa11-41bd-93f3-a761cb50f3ed
2022-11-23T01:20:15.1823812Z GITHUB_JOB=test
2022-11-23T01:20:15.1824063Z SCCACHE_S3_KEY_PREFIX=pull
2022-11-23T01:20:15.1824327Z COMMIT_MESSAGES=
2022-11-23T01:20:15.1824626Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2022-11-23T01:20:15.1824906Z NUM_TEST_SHARDS=3
2022-11-23T01:20:15.1825147Z PR_NUMBER=
2022-11-23T01:20:15.1825704Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_fffd8e99-fa11-41bd-93f3-a761cb50f3ed
2022-11-23T01:20:15.1826109Z SHLVL=1
2022-11-23T01:20:15.1826450Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-6
2022-11-23T01:20:15.1827283Z GITHUB_REPOSITORY=pytorch/pytorch
2022-11-23T01:20:15.1828280Z NVIDIA_REQUIRE_CUDA=cuda>=11.6 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471
2022-11-23T01:20:15.1829126Z NV_LIBNPP_DEV_VERSION=11.6.3.124-1
2022-11-23T01:20:15.1829416Z SHA1=1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T01:20:15.1829698Z GITHUB_EVENT_NAME=push
2022-11-23T01:20:15.1830002Z NV_CUDA_CUDART_VERSION=11.6.55-1
2022-11-23T01:20:15.1830359Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2022-11-23T01:20:15.1830662Z GITHUB_RUN_NUMBER=67324
2022-11-23T01:20:15.1830929Z GITHUB_WORKFLOW=pull
2022-11-23T01:20:15.1831372Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:20:15.1831849Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.12.10-1
2022-11-23T01:20:15.1832314Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T01:20:15.1832699Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2022-11-23T01:20:15.1832970Z _=/usr/bin/env
2022-11-23T01:20:15.1833273Z + echo 'Testing pytorch'
2022-11-23T01:20:15.1833535Z Testing pytorch
2022-11-23T01:20:15.1833800Z + export LANG=C.UTF-8
2022-11-23T01:20:15.1834070Z + LANG=C.UTF-8
2022-11-23T01:20:15.1834311Z + PR_NUMBER=
2022-11-23T01:20:15.1834561Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-11-23T01:20:15.1834872Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-11-23T01:20:15.1835292Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-11-23T01:20:15.1835605Z + [[ distributed == \s\l\o\w ]]
2022-11-23T01:20:15.1836036Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *slow-gradcheck* ]]
2022-11-23T01:20:15.1836497Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-11-23T01:20:15.1836853Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-11-23T01:20:15.1837185Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-11-23T01:20:15.1837492Z + [[ distributed == *crossref* ]]
2022-11-23T01:20:15.1837779Z + [[ distributed == *dynamo* ]]
2022-11-23T01:20:15.1838051Z + [[ distributed == *inductor* ]]
2022-11-23T01:20:15.1838556Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-11-23T01:20:15.1839034Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *-bazel-* ]]
2022-11-23T01:20:15.1839413Z + pip_install --user ninja==1.10.2
2022-11-23T01:20:15.1839831Z + pip install --progress-bar off --user ninja==1.10.2
2022-11-23T01:20:15.7474238Z Collecting ninja==1.10.2
2022-11-23T01:20:15.7669575Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2022-11-23T01:20:16.6394238Z Installing collected packages: ninja
2022-11-23T01:20:16.6495884Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-11-23T01:20:16.6496574Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T01:20:16.6567180Z Successfully installed ninja-1.10.2
2022-11-23T01:20:16.7269036Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:20:16.7269669Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-11-23T01:20:16.7271778Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *asan* ]]
2022-11-23T01:20:16.7272232Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-tsan* ]]
2022-11-23T01:20:16.7272582Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-11-23T01:20:16.7272879Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2022-11-23T01:20:16.7279565Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *tbb* ]]
2022-11-23T01:20:16.7294962Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-11-23T01:20:16.7295396Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-bazel-* ]]
2022-11-23T01:20:16.7295825Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-tsan* ]]
2022-11-23T01:20:16.7298578Z + cd test
2022-11-23T01:20:16.7298973Z + python -c 'import torch; print(torch.__config__.show())'
2022-11-23T01:20:18.4177749Z PyTorch built with:
2022-11-23T01:20:18.4178224Z   - GCC 7.5
2022-11-23T01:20:18.4178512Z   - C++ Version: 201402
2022-11-23T01:20:18.4179045Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-11-23T01:20:18.4179601Z   - Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289)
2022-11-23T01:20:18.4179977Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-11-23T01:20:18.4180347Z   - LAPACK is enabled (usually provided by MKL)
2022-11-23T01:20:18.4180667Z   - NNPACK is enabled
2022-11-23T01:20:18.4180975Z   - CPU capability usage: AVX2
2022-11-23T01:20:18.4181278Z   - CUDA Runtime 11.6
2022-11-23T01:20:18.4181667Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2022-11-23T01:20:18.4182046Z   - CuDNN 8.3.2  (built against CUDA 11.5)
2022-11-23T01:20:18.4182345Z   - Magma 2.6.1
2022-11-23T01:20:18.4185625Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=1.14.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-11-23T01:20:18.4188483Z 
2022-11-23T01:20:18.6525043Z + cd test
2022-11-23T01:20:18.6525583Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-11-23T01:20:20.2614649Z ATen/Parallel:
2022-11-23T01:20:20.2615070Z 	at::get_num_threads() : 16
2022-11-23T01:20:20.2615584Z 	at::get_num_interop_threads() : 16
2022-11-23T01:20:20.2615890Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-11-23T01:20:20.2616150Z 	omp_get_max_threads() : 16
2022-11-23T01:20:20.2616822Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-11-23T01:20:20.2617218Z 	mkl_get_max_threads() : 16
2022-11-23T01:20:20.2617628Z Intel(R) MKL-DNN v2.7.0 (Git Hash 650085b2f3643aad05c629425983491d63b5c289)
2022-11-23T01:20:20.2617982Z std::thread::hardware_concurrency() : 32
2022-11-23T01:20:20.2618283Z Environment variables:
2022-11-23T01:20:20.2618554Z 	OMP_NUM_THREADS : [not set]
2022-11-23T01:20:20.2618806Z 	MKL_NUM_THREADS : [not set]
2022-11-23T01:20:20.2619078Z ATen parallel backend: OpenMP
2022-11-23T01:20:20.2619257Z 
2022-11-23T01:20:20.4872670Z + [[ distributed == *backward* ]]
2022-11-23T01:20:20.4873158Z + [[ distributed == *xla* ]]
2022-11-23T01:20:20.4873454Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-11-23T01:20:20.4874122Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-11-23T01:20:20.4874589Z + [[ distributed == distributed ]]
2022-11-23T01:20:20.4874843Z + install_filelock
2022-11-23T01:20:20.4875596Z + pip_install filelock
2022-11-23T01:20:20.4876049Z + pip install --progress-bar off filelock
2022-11-23T01:20:21.0084712Z Collecting filelock
2022-11-23T01:20:21.0264386Z   Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
2022-11-23T01:20:21.9371832Z Installing collected packages: filelock
2022-11-23T01:20:21.9906121Z Successfully installed filelock-3.8.0
2022-11-23T01:20:22.0555455Z + install_triton
2022-11-23T01:20:22.0555740Z + local commit
2022-11-23T01:20:22.0555985Z + [[ distributed == *rocm* ]]
2022-11-23T01:20:22.0559741Z ++ get_pinned_commit triton
2022-11-23T01:20:22.0560052Z ++ cat .github/ci_commit_pins/triton.txt
2022-11-23T01:20:22.0575064Z + commit=0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:20:22.0575790Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:20:22.0576500Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:20:22.5284199Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2022-11-23T01:20:22.5290623Z   Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-xvfzt2bc
2022-11-23T01:20:22.5311941Z   Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-xvfzt2bc
2022-11-23T01:20:23.2912960Z   Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331'
2022-11-23T01:20:23.2934289Z   Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:20:23.6971887Z   Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:20:23.9224030Z   Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331
2022-11-23T01:20:23.9226674Z   Running command git submodule update --init --recursive -q
2022-11-23T01:20:24.5536874Z   Preparing metadata (setup.py) ... [?25l- done
2022-11-23T01:20:24.7699380Z [?25hCollecting cmake
2022-11-23T01:20:24.7939905Z   Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB)
2022-11-23T01:20:25.2113678Z Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (3.8.0)
2022-11-23T01:20:25.2117196Z Requirement already satisfied: torch in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (1.14.0a0+git1cfd385)
2022-11-23T01:20:25.2379388Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (1.11.1)
2022-11-23T01:20:25.2384465Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (2.6.3)
2022-11-23T01:20:25.2389830Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (4.4.0)
2022-11-23T01:20:25.2608090Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch->triton==2.0.0) (1.2.1)
2022-11-23T01:20:25.2681662Z Building wheels for collected packages: triton
2022-11-23T01:21:19.9232540Z   Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | done
2022-11-23T01:21:19.9715429Z [?25h  Created wheel for triton: filename=triton-2.0.0-cp310-cp310-linux_x86_64.whl size=15377935 sha256=1fa2e08a63150fadd37a6419e417698d5f29ab6c254c4df792a682ed9da33893
2022-11-23T01:21:19.9716403Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/3f/1d/23/1c2bc47d618a44f9c949aea4b7e355e737a1f1ed208f009295
2022-11-23T01:21:19.9736466Z Successfully built triton
2022-11-23T01:21:20.8517110Z Installing collected packages: cmake, triton
2022-11-23T01:21:22.8333639Z Successfully installed cmake-3.25.0 triton-2.0.0
2022-11-23T01:21:22.9416639Z + pip_install --user jinja2
2022-11-23T01:21:22.9417096Z + pip install --progress-bar off --user jinja2
2022-11-23T01:21:23.9708896Z Collecting jinja2
2022-11-23T01:21:23.9896694Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2022-11-23T01:21:24.2567290Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2) (2.1.1)
2022-11-23T01:21:25.1594560Z Installing collected packages: jinja2
2022-11-23T01:21:25.2715202Z Successfully installed jinja2-3.1.2
2022-11-23T01:21:25.3398935Z + test_distributed
2022-11-23T01:21:25.3399419Z + echo 'Testing distributed python tests'
2022-11-23T01:21:25.3399840Z Testing distributed python tests
2022-11-23T01:21:25.3400620Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose
2022-11-23T01:21:27.5470324Z Ignoring disabled issues:  []
2022-11-23T01:21:27.5854297Z /var/lib/jenkins/workspace/test/run_test.py:1134: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-11-23T01:21:27.5854856Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-11-23T01:21:27.5859600Z Found test time stats from artifacts
2022-11-23T01:21:27.5875634Z Selected tests:
2022-11-23T01:21:27.5875952Z  distributed/algorithms/quantization/test_quantization
2022-11-23T01:21:27.5876294Z  distributed/test_distributed_spawn
2022-11-23T01:21:27.5876595Z  distributed/rpc/test_faulty_agent
2022-11-23T01:21:27.5876911Z  distributed/pipeline/sync/test_stream
2022-11-23T01:21:27.5877229Z  distributed/pipeline/sync/test_phony
2022-11-23T01:21:27.5877538Z  distributed/pipeline/sync/test_dependency
2022-11-23T01:21:27.5880108Z  distributed/pipeline/sync/test_checkpoint
2022-11-23T01:21:27.5880490Z  distributed/pipeline/sync/skip/test_verify_skippables
2022-11-23T01:21:27.5880862Z  distributed/pipeline/sync/skip/test_portal
2022-11-23T01:21:27.5881204Z  distributed/pipeline/sync/skip/test_gpipe
2022-11-23T01:21:27.5881498Z  distributed/fsdp/test_shard_utils
2022-11-23T01:21:27.5881800Z  distributed/checkpoint/test_traverse
2022-11-23T01:21:27.5882129Z  distributed/_shard/test_replicated_tensor
2022-11-23T01:21:27.5882416Z  distributed/test_nccl
2022-11-23T01:21:27.5882700Z  distributed/elastic/metrics/api_test
2022-11-23T01:21:27.5883003Z  distributed/nn/jit/test_instantiator
2022-11-23T01:21:27.5883292Z  distributed/elastic/utils/logging_test
2022-11-23T01:21:27.5885612Z  distributed/fsdp/test_checkpoint_wrapper
2022-11-23T01:21:27.5886555Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-11-23T01:21:27.5887329Z  distributed/_tensor/parallel/test_view_sharding_dim_change
2022-11-23T01:21:27.5887698Z  distributed/elastic/timer/local_timer_test
2022-11-23T01:21:27.5888018Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-11-23T01:21:27.5888384Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2022-11-23T01:21:27.5888745Z  distributed/_tensor/parallel/test_2d_parallel
2022-11-23T01:21:27.5889042Z  distributed/_tensor/test_view_ops
2022-11-23T01:21:27.5889374Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T01:21:27.5889728Z  distributed/elastic/timer/local_timer_example
2022-11-23T01:21:27.5890072Z  distributed/_tensor/parallel/test_parallelize_api
2022-11-23T01:21:27.5890415Z  distributed/_shard/sharding_plan/test_sharding_plan
2022-11-23T01:21:27.5890730Z  distributed/_tensor/test_api
2022-11-23T01:21:27.5891026Z  distributed/fsdp/test_fsdp_overlap
2022-11-23T01:21:27.5891325Z  distributed/test_c10d_object_collectives
2022-11-23T01:21:27.5891635Z  distributed/test_dynamo_distributed
2022-11-23T01:21:27.5891948Z  distributed/_composable/test_fully_shard
2022-11-23T01:21:27.5892256Z  distributed/fsdp/test_fsdp_ignored_modules
2022-11-23T01:21:27.5892592Z  distributed/elastic/multiprocessing/api_test
2022-11-23T01:21:27.5892909Z  distributed/fsdp/test_fsdp_meta
2022-11-23T01:21:27.5893222Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2022-11-23T01:21:27.5893552Z  distributed/fsdp/test_fsdp_exec_order
2022-11-23T01:21:27.5893852Z  distributed/_tensor/test_dtensor
2022-11-23T01:21:27.5894122Z  distributed/algorithms/test_join
2022-11-23T01:21:27.5894566Z  distributed/fsdp/test_fsdp_grad_acc
2022-11-23T01:21:27.5894865Z  distributed/fsdp/test_fsdp_misc
2022-11-23T01:21:27.5895149Z  distributed/fsdp/test_fsdp_checkpoint
2022-11-23T01:21:27.5895452Z  distributed/test_c10d_pypg
2022-11-23T01:21:27.5895755Z  distributed/fsdp/test_fsdp_use_orig_params
2022-11-23T01:21:27.5896082Z  distributed/fsdp/test_fsdp_mixed_precision
2022-11-23T01:21:27.5896391Z  distributed/rpc/cuda/test_tensorpipe_agent
2022-11-23T01:21:27.5900086Z Prioritized test from test file changes.
2022-11-23T01:21:27.5900651Z reordering tests for PR:
2022-11-23T01:21:27.5900918Z prioritized: []
2022-11-23T01:21:27.5905524Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/rpc/test_faulty_agent', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_phony', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_checkpoint', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/skip/test_portal', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/fsdp/test_shard_utils', 'distributed/checkpoint/test_traverse', 'distributed/_shard/test_replicated_tensor', 'distributed/test_nccl', 'distributed/elastic/metrics/api_test', 'distributed/nn/jit/test_instantiator', 'distributed/elastic/utils/logging_test', 'distributed/fsdp/test_checkpoint_wrapper', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/_tensor/parallel/test_view_sharding_dim_change', 'distributed/elastic/timer/local_timer_test', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard', 'distributed/_tensor/parallel/test_2d_parallel', 'distributed/_tensor/test_view_ops', 'distributed/_shard/sharded_tensor/ops/test_linear', 'distributed/elastic/timer/local_timer_example', 'distributed/_tensor/parallel/test_parallelize_api', 'distributed/_shard/sharding_plan/test_sharding_plan', 'distributed/_tensor/test_api', 'distributed/fsdp/test_fsdp_overlap', 'distributed/test_c10d_object_collectives', 'distributed/test_dynamo_distributed', 'distributed/_composable/test_fully_shard', 'distributed/fsdp/test_fsdp_ignored_modules', 'distributed/elastic/multiprocessing/api_test', 'distributed/fsdp/test_fsdp_meta', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks', 'distributed/fsdp/test_fsdp_exec_order', 'distributed/_tensor/test_dtensor', 'distributed/algorithms/test_join', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/fsdp/test_fsdp_misc', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/test_c10d_pypg', 'distributed/fsdp/test_fsdp_use_orig_params', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/rpc/cuda/test_tensorpipe_agent']
2022-11-23T01:21:27.5908790Z 
2022-11-23T01:21:27.5909344Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-11-23T01:21:27.6119141Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-11-23T01:21:27.6276454Z parallel (file granularity) tests:
2022-11-23T01:21:27.6276927Z  
2022-11-23T01:21:27.6277373Z serial (file granularity) tests:
2022-11-23T01:21:27.6277902Z  distributed/algorithms/quantization/test_quantization
2022-11-23T01:21:27.6278494Z  distributed/test_distributed_spawn
2022-11-23T01:21:27.6279054Z  distributed/rpc/test_faulty_agent
2022-11-23T01:21:27.6279647Z  distributed/pipeline/sync/test_stream
2022-11-23T01:21:27.6280220Z  distributed/pipeline/sync/test_phony
2022-11-23T01:21:27.6280684Z  distributed/pipeline/sync/test_dependency
2022-11-23T01:21:27.6281015Z  distributed/pipeline/sync/test_checkpoint
2022-11-23T01:21:27.6281356Z  distributed/pipeline/sync/skip/test_verify_skippables
2022-11-23T01:21:27.6281684Z  distributed/pipeline/sync/skip/test_portal
2022-11-23T01:21:27.6282009Z  distributed/pipeline/sync/skip/test_gpipe
2022-11-23T01:21:27.6282492Z  distributed/fsdp/test_shard_utils
2022-11-23T01:21:27.6282778Z  distributed/checkpoint/test_traverse
2022-11-23T01:21:27.6283094Z  distributed/_shard/test_replicated_tensor
2022-11-23T01:21:27.6283384Z  distributed/test_nccl
2022-11-23T01:21:27.6283649Z  distributed/elastic/metrics/api_test
2022-11-23T01:21:27.6283961Z  distributed/nn/jit/test_instantiator
2022-11-23T01:21:27.6284265Z  distributed/elastic/utils/logging_test
2022-11-23T01:21:27.6284564Z  distributed/fsdp/test_checkpoint_wrapper
2022-11-23T01:21:27.6284916Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2022-11-23T01:21:27.6285292Z  distributed/_tensor/parallel/test_view_sharding_dim_change
2022-11-23T01:21:27.6285635Z  distributed/elastic/timer/local_timer_test
2022-11-23T01:21:27.6285947Z  distributed/fsdp/test_fsdp_multiple_wrapping
2022-11-23T01:21:27.6286312Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2022-11-23T01:21:27.6286667Z  distributed/_tensor/parallel/test_2d_parallel
2022-11-23T01:21:27.6286967Z  distributed/_tensor/test_view_ops
2022-11-23T01:21:27.6287286Z  distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T01:21:27.6287623Z  distributed/elastic/timer/local_timer_example
2022-11-23T01:21:27.6287948Z  distributed/_tensor/parallel/test_parallelize_api
2022-11-23T01:21:27.6288302Z  distributed/_shard/sharding_plan/test_sharding_plan
2022-11-23T01:21:27.6288641Z  distributed/_tensor/test_api
2022-11-23T01:21:27.6288935Z  distributed/fsdp/test_fsdp_overlap
2022-11-23T01:21:27.6289225Z  distributed/test_c10d_object_collectives
2022-11-23T01:21:27.6289529Z  distributed/test_dynamo_distributed
2022-11-23T01:21:27.6289839Z  distributed/_composable/test_fully_shard
2022-11-23T01:21:27.6290147Z  distributed/fsdp/test_fsdp_ignored_modules
2022-11-23T01:21:27.6290479Z  distributed/elastic/multiprocessing/api_test
2022-11-23T01:21:27.6290791Z  distributed/fsdp/test_fsdp_meta
2022-11-23T01:21:27.6291100Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2022-11-23T01:21:27.6291434Z  distributed/fsdp/test_fsdp_exec_order
2022-11-23T01:21:27.6291730Z  distributed/_tensor/test_dtensor
2022-11-23T01:21:27.6292002Z  distributed/algorithms/test_join
2022-11-23T01:21:27.6292297Z  distributed/fsdp/test_fsdp_grad_acc
2022-11-23T01:21:27.6292591Z  distributed/fsdp/test_fsdp_misc
2022-11-23T01:21:27.6292871Z  distributed/fsdp/test_fsdp_checkpoint
2022-11-23T01:21:27.6293246Z  distributed/test_c10d_pypg
2022-11-23T01:21:27.6293562Z  distributed/fsdp/test_fsdp_use_orig_params
2022-11-23T01:21:27.6293869Z  distributed/fsdp/test_fsdp_mixed_precision
2022-11-23T01:21:27.6294193Z  distributed/rpc/cuda/test_tensorpipe_agent
2022-11-23T01:21:29.8356652Z Ignoring disabled issues:  []
2022-11-23T01:21:29.8621379Z Ignoring disabled issues:  []
2022-11-23T01:21:30.2705209Z Running distributed/algorithms/quantization/test_quantization ... [2022-11-23 01:21:30.270023]
2022-11-23T01:21:30.2714062Z /usr/bin/mpiexec
2022-11-23T01:21:30.2714569Z MPI not available -- MPI backend tests will be skipped
2022-11-23T01:21:30.2715194Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2}
2022-11-23T01:21:30.2719212Z Running distributed tests for the test backend with env init_method in shard 1 of 3
2022-11-23T01:21:30.2724606Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:21:30.272156]
2022-11-23T01:21:32.3787590Z 
2022-11-23T01:21:32.3788304Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:21:32.3794146Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_fwejmzht)
2022-11-23T01:21:32.3794762Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:21:32.3794965Z 
2022-11-23T01:21:32.3795581Z ##[endgroup]
2022-11-23T01:21:32.3796400Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_fwejmzht)
2022-11-23T01:21:32.3796842Z 
2022-11-23T01:21:32.3798968Z Running distributed tests for the test backend with file init_method in shard 1 of 3
2022-11-23T01:21:32.3803453Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:21:32.380020]
2022-11-23T01:21:34.4868455Z 
2022-11-23T01:21:34.4869255Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:21:34.4870443Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_z5bufbgd)
2022-11-23T01:21:34.4871051Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:21:34.4871252Z 
2022-11-23T01:21:34.4871535Z ##[endgroup]
2022-11-23T01:21:34.4872372Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_z5bufbgd)
2022-11-23T01:21:34.4872799Z 
2022-11-23T01:21:34.4878016Z Shard 1: nccl should be run in 2
2022-11-23T01:21:34.4878722Z Running distributed tests for the gloo backend with env init_method in shard 1 of 3
2022-11-23T01:21:34.4883718Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:21:34.488070]
2022-11-23T01:21:57.4383847Z 
2022-11-23T01:21:57.4384358Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:21:57.4387936Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_ch8f2149)
2022-11-23T01:21:57.4389351Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-11-23T01:21:57.4390298Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:21:57.4390654Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:21:57.4391012Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:21:57.4391373Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:21:57.4391727Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:21:57.4392122Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:21:57.4392493Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:21:57.4393168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:21:57.4393626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:21:57.4394214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:21:57.4395303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:21:57.4395557Z 
2022-11-23T01:21:57.4395650Z Running tests...
2022-11-23T01:21:57.4396077Z ----------------------------------------------------------------------
2022-11-23T01:21:57.4396686Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:21:57.4397238Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:21:57.4397886Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 976
2022-11-23T01:21:57.4398329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 977
2022-11-23T01:21:57.4399757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:21:57.4400636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:21:57.4401435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:21:57.4401919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:21:57.4402496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:21:57.4402951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:21:57.4403535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:21:57.4403996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:21:57.4404764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:21:57.4405417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:21:57.4406326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:21:57.4407409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:21:57.4408739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:21:57.4410255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:21:57.4411117Z ok (3.930s)
2022-11-23T01:21:57.4411384Z 
2022-11-23T01:21:57.4411953Z ----------------------------------------------------------------------
2022-11-23T01:21:57.4412637Z Ran 1 test in 3.930s
2022-11-23T01:21:57.4412978Z 
2022-11-23T01:21:57.4413154Z OK
2022-11-23T01:21:57.4413388Z 
2022-11-23T01:21:57.4413520Z Generating XML reports...
2022-11-23T01:21:57.4414315Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012138.xml
2022-11-23T01:21:57.4415097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:21:57.4415544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:21:57.4416111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:21:57.4416563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:21:57.4416797Z 
2022-11-23T01:21:57.4416904Z Running tests...
2022-11-23T01:21:57.4417303Z ----------------------------------------------------------------------
2022-11-23T01:21:57.4417884Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:21:57.4418426Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:21:57.4418897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1085
2022-11-23T01:21:57.4419340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1086
2022-11-23T01:21:57.4419927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:21:57.4420434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:21:57.4421061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:21:57.4421612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:21:57.4422176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:21:57.4422618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:21:57.4423187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:21:57.4423647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:21:57.4424058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:21:57.4424527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:21:57.4425006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:21:57.4425498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:21:57.4426137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:21:57.4427535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:21:57.4428095Z ok (3.915s)
2022-11-23T01:21:57.4428244Z 
2022-11-23T01:21:57.4428506Z ----------------------------------------------------------------------
2022-11-23T01:21:57.4428835Z Ran 1 test in 3.916s
2022-11-23T01:21:57.4428995Z 
2022-11-23T01:21:57.4429086Z OK
2022-11-23T01:21:57.4429216Z 
2022-11-23T01:21:57.4429337Z Generating XML reports...
2022-11-23T01:21:57.4429982Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012144.xml
2022-11-23T01:21:57.4430735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:21:57.4431185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:21:57.4431742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:21:57.4432341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:21:57.4432587Z 
2022-11-23T01:21:57.4432693Z Running tests...
2022-11-23T01:21:57.4433098Z ----------------------------------------------------------------------
2022-11-23T01:21:57.4433678Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:21:57.4434236Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-11-23T01:21:57.4434527Z 
2022-11-23T01:21:57.4434779Z ----------------------------------------------------------------------
2022-11-23T01:21:57.4435102Z Ran 1 test in 0.001s
2022-11-23T01:21:57.4435244Z 
2022-11-23T01:21:57.4435349Z OK (skipped=1)
2022-11-23T01:21:57.4435502Z 
2022-11-23T01:21:57.4435622Z Generating XML reports...
2022-11-23T01:21:57.4436276Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012150.xml
2022-11-23T01:21:57.4437023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:21:57.4437459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:21:57.4438031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:21:57.4438497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:21:57.4438707Z 
2022-11-23T01:21:57.4438813Z Running tests...
2022-11-23T01:21:57.4439209Z ----------------------------------------------------------------------
2022-11-23T01:21:57.4439902Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:21:57.4440445Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-11-23T01:21:57.4440728Z 
2022-11-23T01:21:57.4440990Z ----------------------------------------------------------------------
2022-11-23T01:21:57.4441314Z Ran 1 test in 0.001s
2022-11-23T01:21:57.4441473Z 
2022-11-23T01:21:57.4441579Z OK (skipped=1)
2022-11-23T01:21:57.4441716Z 
2022-11-23T01:21:57.4441837Z Generating XML reports...
2022-11-23T01:21:57.4442488Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012152.xml
2022-11-23T01:21:57.4443233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:21:57.4443678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:21:57.4444235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:21:57.4444700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:21:57.4444926Z 
2022-11-23T01:21:57.4445031Z Running tests...
2022-11-23T01:21:57.4445417Z ----------------------------------------------------------------------
2022-11-23T01:21:57.4446020Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:21:57.4446597Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s)
2022-11-23T01:21:57.4446896Z 
2022-11-23T01:21:57.4447155Z ----------------------------------------------------------------------
2022-11-23T01:21:57.4447457Z Ran 1 test in 0.001s
2022-11-23T01:21:57.4447615Z 
2022-11-23T01:21:57.4447719Z OK (skipped=1)
2022-11-23T01:21:57.4447875Z 
2022-11-23T01:21:57.4447996Z Generating XML reports...
2022-11-23T01:21:57.4448633Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012154.xml
2022-11-23T01:21:57.4449388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:21:57.4449893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:21:57.4450476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:21:57.4450921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:21:57.4451149Z 
2022-11-23T01:21:57.4451255Z Running tests...
2022-11-23T01:21:57.4451650Z ----------------------------------------------------------------------
2022-11-23T01:21:57.4452230Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:21:57.4452816Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s)
2022-11-23T01:21:57.4453110Z 
2022-11-23T01:21:57.4453371Z ----------------------------------------------------------------------
2022-11-23T01:21:57.4453692Z Ran 1 test in 0.001s
2022-11-23T01:21:57.4453851Z 
2022-11-23T01:21:57.4453944Z OK (skipped=1)
2022-11-23T01:21:57.4454096Z 
2022-11-23T01:21:57.4454216Z Generating XML reports...
2022-11-23T01:21:57.4454872Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012156.xml
2022-11-23T01:21:57.4455268Z 
2022-11-23T01:21:57.4455710Z ##[endgroup]
2022-11-23T01:21:57.4456410Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_ch8f2149)
2022-11-23T01:21:57.4456932Z 
2022-11-23T01:21:57.4457147Z Running distributed tests for the gloo backend with file init_method in shard 1 of 3
2022-11-23T01:21:57.4457925Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:21:57.439833]
2022-11-23T01:22:20.5053003Z 
2022-11-23T01:22:20.5053522Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:22:20.5056621Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_kjse4vca)
2022-11-23T01:22:20.5057764Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2022-11-23T01:22:20.5058661Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:20.5059013Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:20.5059386Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:20.5059746Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:20.5060107Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:20.5060485Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2022-11-23T01:22:20.5060834Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:22:20.5061497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:20.5061961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:20.5062549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:20.5064339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:20.5064672Z 
2022-11-23T01:22:20.5064793Z Running tests...
2022-11-23T01:22:20.5065489Z ----------------------------------------------------------------------
2022-11-23T01:22:20.5066127Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:20.5066664Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:22:20.5067899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1359
2022-11-23T01:22:20.5068344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1360
2022-11-23T01:22:20.5069014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:20.5069465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:20.5070045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:20.5070523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:20.5071119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:20.5071541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:20.5072119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:20.5072575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:20.5073009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:22:20.5073486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:22:20.5074129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:22:20.5074636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:22:20.5075271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:20.5075983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:20.5076421Z ok (3.966s)
2022-11-23T01:22:20.5076561Z 
2022-11-23T01:22:20.5076826Z ----------------------------------------------------------------------
2022-11-23T01:22:20.5077154Z Ran 1 test in 3.966s
2022-11-23T01:22:20.5077296Z 
2022-11-23T01:22:20.5077388Z OK
2022-11-23T01:22:20.5077520Z 
2022-11-23T01:22:20.5077663Z Generating XML reports...
2022-11-23T01:22:20.5078308Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012201.xml
2022-11-23T01:22:20.5079055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:20.5079517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:20.5080090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:20.5080555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:20.5080766Z 
2022-11-23T01:22:20.5080872Z Running tests...
2022-11-23T01:22:20.5081285Z ----------------------------------------------------------------------
2022-11-23T01:22:20.5081865Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:20.5082409Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:22:20.5082854Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1468
2022-11-23T01:22:20.5083294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1469
2022-11-23T01:22:20.5083991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:20.5084435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:20.5085007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:20.5085475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:20.5086045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:20.5086468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:20.5087032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:20.5087490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:20.5087906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:22:20.5088377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:22:20.5088863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:22:20.5089353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:22:20.5089992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:20.5090676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:22:20.5091138Z ok (3.948s)
2022-11-23T01:22:20.5091285Z 
2022-11-23T01:22:20.5091536Z ----------------------------------------------------------------------
2022-11-23T01:22:20.5091861Z Ran 1 test in 3.948s
2022-11-23T01:22:20.5092022Z 
2022-11-23T01:22:20.5092114Z OK
2022-11-23T01:22:20.5092247Z 
2022-11-23T01:22:20.5092369Z Generating XML reports...
2022-11-23T01:22:20.5093031Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012207.xml
2022-11-23T01:22:20.5093770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:20.5094233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:20.5094778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:20.5095246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:20.5095498Z 
2022-11-23T01:22:20.5095585Z Running tests...
2022-11-23T01:22:20.5095984Z ----------------------------------------------------------------------
2022-11-23T01:22:20.5096563Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:20.5097123Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-11-23T01:22:20.5097408Z 
2022-11-23T01:22:20.5097670Z ----------------------------------------------------------------------
2022-11-23T01:22:20.5097991Z Ran 1 test in 0.001s
2022-11-23T01:22:20.5098132Z 
2022-11-23T01:22:20.5098238Z OK (skipped=1)
2022-11-23T01:22:20.5098389Z 
2022-11-23T01:22:20.5098509Z Generating XML reports...
2022-11-23T01:22:20.5099158Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012213.xml
2022-11-23T01:22:20.5099893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:20.5100339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:20.5100910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:20.5101457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:20.5101673Z 
2022-11-23T01:22:20.5101782Z Running tests...
2022-11-23T01:22:20.5102183Z ----------------------------------------------------------------------
2022-11-23T01:22:20.5102780Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:20.5103319Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2022-11-23T01:22:20.5103607Z 
2022-11-23T01:22:20.5103892Z ----------------------------------------------------------------------
2022-11-23T01:22:20.5104198Z Ran 1 test in 0.001s
2022-11-23T01:22:20.5104356Z 
2022-11-23T01:22:20.5104461Z OK (skipped=1)
2022-11-23T01:22:20.5104596Z 
2022-11-23T01:22:20.5104717Z Generating XML reports...
2022-11-23T01:22:20.5105370Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012215.xml
2022-11-23T01:22:20.5106139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:20.5106548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:20.5107792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:20.5108260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:20.5108490Z 
2022-11-23T01:22:20.5108598Z Running tests...
2022-11-23T01:22:20.5108984Z ----------------------------------------------------------------------
2022-11-23T01:22:20.5109702Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:20.5110280Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s)
2022-11-23T01:22:20.5110581Z 
2022-11-23T01:22:20.5110845Z ----------------------------------------------------------------------
2022-11-23T01:22:20.5111149Z Ran 1 test in 0.001s
2022-11-23T01:22:20.5111307Z 
2022-11-23T01:22:20.5111413Z OK (skipped=1)
2022-11-23T01:22:20.5111566Z 
2022-11-23T01:22:20.5111688Z Generating XML reports...
2022-11-23T01:22:20.5112328Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012217.xml
2022-11-23T01:22:20.5113085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:22:20.5113527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:22:20.5114107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:22:20.5114554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:22:20.5114805Z 
2022-11-23T01:22:20.5114893Z Running tests...
2022-11-23T01:22:20.5115294Z ----------------------------------------------------------------------
2022-11-23T01:22:20.5115880Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2022-11-23T01:22:20.5116453Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s)
2022-11-23T01:22:20.5116755Z 
2022-11-23T01:22:20.5117016Z ----------------------------------------------------------------------
2022-11-23T01:22:20.5117337Z Ran 1 test in 0.001s
2022-11-23T01:22:20.5117494Z 
2022-11-23T01:22:20.5117582Z OK (skipped=1)
2022-11-23T01:22:20.5117737Z 
2022-11-23T01:22:20.5117857Z Generating XML reports...
2022-11-23T01:22:20.5118509Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012219.xml
2022-11-23T01:22:20.5118910Z 
2022-11-23T01:22:20.5119338Z ##[endgroup]
2022-11-23T01:22:20.5120170Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_kjse4vca)
2022-11-23T01:22:20.5120619Z 
2022-11-23T01:22:20.5120828Z Running distributed tests for the ucc backend with env init_method in shard 1 of 3
2022-11-23T01:22:20.5121612Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:22:20.506795]
2022-11-23T01:22:22.5639473Z 
2022-11-23T01:22:22.5640135Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:22:22.5641193Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization__z386hrq)
2022-11-23T01:22:22.5641769Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:22:22.5641971Z 
2022-11-23T01:22:22.5642256Z ##[endgroup]
2022-11-23T01:22:22.5643069Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization__z386hrq)
2022-11-23T01:22:22.5643504Z 
2022-11-23T01:22:22.5651369Z Running distributed tests for the ucc backend with file init_method in shard 1 of 3
2022-11-23T01:22:22.5656674Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:22:22.565365]
2022-11-23T01:22:24.6765424Z 
2022-11-23T01:22:24.6766194Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2022-11-23T01:22:24.6767361Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_ky1p9z6c)
2022-11-23T01:22:24.6767957Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:22:24.6768159Z 
2022-11-23T01:22:24.6768451Z ##[endgroup]
2022-11-23T01:22:24.6769289Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_ky1p9z6c)
2022-11-23T01:22:24.6769728Z 
2022-11-23T01:22:24.6778796Z Running distributed/test_distributed_spawn ... [2022-11-23 01:22:24.677545]
2022-11-23T01:22:24.6787437Z /usr/bin/mpiexec
2022-11-23T01:22:24.6788288Z MPI not available -- MPI backend tests will be skipped
2022-11-23T01:22:24.6788873Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 3}
2022-11-23T01:22:24.6790709Z Running distributed tests for the test backend with env init_method in shard 1 of 3
2022-11-23T01:22:24.6795213Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:22:24.679231]
2022-11-23T01:22:27.0902245Z 
2022-11-23T01:22:27.0902932Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T01:22:27.0903848Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_6433x6oy)
2022-11-23T01:22:27.0904315Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:22:27.0904513Z 
2022-11-23T01:22:27.0904817Z ##[endgroup]
2022-11-23T01:22:27.0905529Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_6433x6oy)
2022-11-23T01:22:27.0905898Z 
2022-11-23T01:22:27.0913864Z Running distributed tests for the test backend with file init_method in shard 1 of 3
2022-11-23T01:22:27.0918807Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:22:27.091538]
2022-11-23T01:22:29.4690469Z 
2022-11-23T01:22:29.4691174Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T01:22:29.4692121Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_hb_2sj3a)
2022-11-23T01:22:29.4692618Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:22:29.4692816Z 
2022-11-23T01:22:29.4693091Z ##[endgroup]
2022-11-23T01:22:29.4693788Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_hb_2sj3a)
2022-11-23T01:22:29.4694153Z 
2022-11-23T01:22:29.4700010Z Shard 1: nccl should be run in 2
2022-11-23T01:22:29.4700652Z Running distributed tests for the gloo backend with env init_method in shard 1 of 3
2022-11-23T01:22:29.4706205Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:22:29.470312]
2022-11-23T01:51:49.0447751Z 
2022-11-23T01:51:49.0448199Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T01:51:49.0449129Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_oa_8p1ar)
2022-11-23T01:51:49.0457592Z <unittest.suite.TestSuite tests=[]>
2022-11-23T01:51:49.0517899Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-11-23T01:51:49.0573414Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0573914Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0589910Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0590732Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0591598Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0592090Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0592582Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0593078Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0593606Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0594132Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0594859Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0595559Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0596098Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0596603Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0597085Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0597577Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0598032Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0598473Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0598907Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0599360Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0599832Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0600310Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0600724Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0601113Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0601548Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0601967Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0602389Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0602800Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0603211Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0603590Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0603976Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0604382Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0604777Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0605182Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0605676Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0606108Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0606519Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0606921Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0607342Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0607746Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0608141Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0608601Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0609048Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0609498Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0609921Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0610349Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0610784Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0611198Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0611615Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0612059Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0612502Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0612965Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0613375Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0613802Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0614219Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0614627Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0615048Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0615466Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0615845Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0616240Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0616643Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0617035Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0617419Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0617788Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0618177Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0618576Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0618981Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0619376Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0619745Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0620128Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0620517Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0620895Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0621327Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0621745Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0622133Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0622492Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0622873Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0623317Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0623710Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0624112Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0624512Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0624879Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0625287Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0625719Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0626163Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0626593Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0627715Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0628186Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0628624Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0629075Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0629512Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0629947Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0630381Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0630941Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0631403Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0631854Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0632334Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0632789Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0633215Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0633589Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0633974Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0634343Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0634690Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0635080Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0635490Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0635880Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0636242Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0636648Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0637059Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0637441Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0637839Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0638249Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0638660Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0639083Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0639503Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0639933Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0640331Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0640763Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0641264Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0641675Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0642067Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0642441Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0642832Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0643201Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0643594Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0644004Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0644442Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0644963Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0645424Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0645848Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0646246Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0646682Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0647133Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0647590Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0648031Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0648537Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0648981Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0649368Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0649748Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0650142Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0650578Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0650994Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0651435Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0651856Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0652275Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0652695Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0653161Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0653649Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0654215Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0654833Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0655433Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0656009Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0656609Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0657211Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0657856Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0658451Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0658991Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0659483Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0659921Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0660294Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0660685Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0661085Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0661462Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0661881Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0662319Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0662777Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0663222Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0663638Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0664022Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0664479Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0664916Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0665343Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0665739Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0666152Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0666575Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0667651Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0668060Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0668465Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0668868Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0669267Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0669675Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0670093Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0670549Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0670968Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0671426Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0671805Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0672188Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0672602Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0673016Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0673379Z test_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0673746Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0674111Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0674489Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0674845Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0675211Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0675687Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0676070Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0676429Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0676781Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0677145Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0677540Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0677927Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0678302Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0678627Z test_isend (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0678999Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0679398Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0679795Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0680248Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0680697Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0681091Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0681505Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0681935Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0682358Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0682844Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0683256Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0683666Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0684059Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0684464Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0684854Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0685232Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0685697Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0686182Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0686647Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0687086Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0687546Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0687998Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0688410Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0688878Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0689305Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0689738Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0690166Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0690639Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0691151Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0691616Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0692031Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0692422Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0692884Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0693282Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0693665Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0694042Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0694415Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0694800Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0695167Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0695519Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0695894Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0696270Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0696664Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0697053Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0697427Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0697796Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0698163Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0698546Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0698911Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0699299Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0699676Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0700111Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0700491Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0700863Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0701237Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0701617Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0701968Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0702339Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0702758Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0703182Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0703609Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0704005Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0704408Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0704816Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0705227Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0705617Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0706013Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0706446Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0707507Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0707977Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0708365Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0708761Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0709141Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0709549Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0709994Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0710436Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.0711266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0711744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0712331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0712799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0713030Z 
2022-11-23T01:51:49.0713122Z Running tests...
2022-11-23T01:51:49.0713527Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0714061Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0714645Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0715184Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1875
2022-11-23T01:51:49.0715631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1876
2022-11-23T01:51:49.0716242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0716675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0717249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0717716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0718390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0718815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0719385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0719851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0720288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.0720778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.0721439Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0722127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0722639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.0723103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.0723628Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:51:49.0724466Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:51:49.0725116Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:51:49.0725929Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:51:49.0726577Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:51:49.0727141Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:51:49.0727939Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:51:49.0728913Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:51:49.0729578Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:51:49.0730144Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:51:49.0730948Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:51:49.0731861Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:51:49.0732511Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:51:49.0733072Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T01:51:49.0733875Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:51:49.0734772Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T01:51:49.0735252Z ok (6.958s)
2022-11-23T01:51:49.0735400Z 
2022-11-23T01:51:49.0735740Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0736047Z Ran 1 test in 6.958s
2022-11-23T01:51:49.0736205Z 
2022-11-23T01:51:49.0736297Z OK
2022-11-23T01:51:49.0736429Z 
2022-11-23T01:51:49.0736551Z Generating XML reports...
2022-11-23T01:51:49.0737168Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012233.xml
2022-11-23T01:51:49.0737873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0738318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0738889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0739337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0739567Z 
2022-11-23T01:51:49.0739673Z Running tests...
2022-11-23T01:51:49.0740079Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0740605Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0741116Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-11-23T01:51:49.0741425Z 
2022-11-23T01:51:49.0741693Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0742014Z Ran 1 test in 0.004s
2022-11-23T01:51:49.0742171Z 
2022-11-23T01:51:49.0742261Z OK (skipped=1)
2022-11-23T01:51:49.0742411Z 
2022-11-23T01:51:49.0742532Z Generating XML reports...
2022-11-23T01:51:49.0743136Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012243.xml
2022-11-23T01:51:49.0743847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0744282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0744850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0745313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0745540Z 
2022-11-23T01:51:49.0745646Z Running tests...
2022-11-23T01:51:49.0746087Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0746628Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0747595Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0748065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2020
2022-11-23T01:51:49.0748506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2021
2022-11-23T01:51:49.0749119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0749573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0750133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0750601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0751179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0751599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0752169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0752629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0753078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.0753663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.0754318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0755009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0755532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.0755984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.0756318Z ok (4.153s)
2022-11-23T01:51:49.0756465Z 
2022-11-23T01:51:49.0756735Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0757044Z Ran 1 test in 4.153s
2022-11-23T01:51:49.0757201Z 
2022-11-23T01:51:49.0757294Z OK
2022-11-23T01:51:49.0757428Z 
2022-11-23T01:51:49.0757555Z Generating XML reports...
2022-11-23T01:51:49.0758145Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012245.xml
2022-11-23T01:51:49.0758858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0759306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0759880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0760335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0760564Z 
2022-11-23T01:51:49.0760671Z Running tests...
2022-11-23T01:51:49.0761070Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0761600Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0762109Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0763236Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.647s)
2022-11-23T01:51:49.0763768Z 
2022-11-23T01:51:49.0764038Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0764362Z Ran 1 test in 1.647s
2022-11-23T01:51:49.0764521Z 
2022-11-23T01:51:49.0764610Z OK (skipped=1)
2022-11-23T01:51:49.0764760Z 
2022-11-23T01:51:49.0764882Z Generating XML reports...
2022-11-23T01:51:49.0765485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012252.xml
2022-11-23T01:51:49.0766195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0766634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0802670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0803159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0803406Z 
2022-11-23T01:51:49.0803516Z Running tests...
2022-11-23T01:51:49.0803928Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0804446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0804993Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0805506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2163
2022-11-23T01:51:49.0805951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2164
2022-11-23T01:51:49.0806719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0807173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0807756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0808230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0808794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0809232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0809802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0810249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0810703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.0811200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.0811856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0812538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0813057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.0813554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyr2_w_zp
2022-11-23T01:51:49.0814097Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyr2_w_zp/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0814586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.0815090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaaywyb6r
2022-11-23T01:51:49.0815622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaaywyb6r/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0816119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0816682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0817054Z ok (4.359s)
2022-11-23T01:51:49.0817206Z 
2022-11-23T01:51:49.0817488Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0817799Z Ran 1 test in 4.359s
2022-11-23T01:51:49.0817958Z 
2022-11-23T01:51:49.0818054Z OK
2022-11-23T01:51:49.0818189Z 
2022-11-23T01:51:49.0818310Z Generating XML reports...
2022-11-23T01:51:49.0818899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012256.xml
2022-11-23T01:51:49.0819622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0820067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0820641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0821095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0821326Z 
2022-11-23T01:51:49.0821436Z Running tests...
2022-11-23T01:51:49.0821838Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0822351Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0822904Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0823433Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2276
2022-11-23T01:51:49.0823955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2277
2022-11-23T01:51:49.0824546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0824993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0825573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0826021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0826603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0827478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0828066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0828521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0828977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.0829474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.0830138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0830813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0831337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.0831810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.0832293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbu8iipn
2022-11-23T01:51:49.0832830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuflnofe5
2022-11-23T01:51:49.0833365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbu8iipn/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0833905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuflnofe5/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0834523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0835025Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0835378Z ok (4.247s)
2022-11-23T01:51:49.0835529Z 
2022-11-23T01:51:49.0835789Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0836118Z Ran 1 test in 4.247s
2022-11-23T01:51:49.0836279Z 
2022-11-23T01:51:49.0836375Z OK
2022-11-23T01:51:49.0836510Z 
2022-11-23T01:51:49.0836634Z Generating XML reports...
2022-11-23T01:51:49.0837221Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012303.xml
2022-11-23T01:51:49.0837944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0838391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0838953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0839422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0839651Z 
2022-11-23T01:51:49.0839758Z Running tests...
2022-11-23T01:51:49.0840160Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0840671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0841218Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0841828Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2389
2022-11-23T01:51:49.0842279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2390
2022-11-23T01:51:49.0842870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0843323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0843901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0844354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0844930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0845373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0845934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0846385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0846836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.0847342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.0847980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0848664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0849189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.0849660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.0850139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqxoe0l6
2022-11-23T01:51:49.0850682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqxoe0l6/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0851210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbppv0gbl
2022-11-23T01:51:49.0851794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbppv0gbl/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0852306Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0852791Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0853268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0853730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0854203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0854675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0855146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0855598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0856068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0856535Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0856987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0857463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0857804Z ok (6.842s)
2022-11-23T01:51:49.0857950Z 
2022-11-23T01:51:49.0858230Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0858604Z Ran 1 test in 6.842s
2022-11-23T01:51:49.0858765Z 
2022-11-23T01:51:49.0858857Z OK
2022-11-23T01:51:49.0858989Z 
2022-11-23T01:51:49.0859111Z Generating XML reports...
2022-11-23T01:51:49.0859704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012310.xml
2022-11-23T01:51:49.0860422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0860868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0861444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0861896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0862124Z 
2022-11-23T01:51:49.0862231Z Running tests...
2022-11-23T01:51:49.0862627Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0863141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0863701Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0864235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2504
2022-11-23T01:51:49.0864684Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2505
2022-11-23T01:51:49.0865276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0865724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0866286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0866731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0867644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0868119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0868707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0869252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0869727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.0870227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.0870889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0871564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0872093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.0872560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.0873058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph537q39w
2022-11-23T01:51:49.0873580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph537q39w/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0874109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp157f6sfh
2022-11-23T01:51:49.0874640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp157f6sfh/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0875130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0875612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0876087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0876652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0876981Z ok (5.750s)
2022-11-23T01:51:49.0877126Z 
2022-11-23T01:51:49.0877400Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0877731Z Ran 1 test in 5.750s
2022-11-23T01:51:49.0877893Z 
2022-11-23T01:51:49.0877972Z OK
2022-11-23T01:51:49.0878108Z 
2022-11-23T01:51:49.0878235Z Generating XML reports...
2022-11-23T01:51:49.0878851Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012319.xml
2022-11-23T01:51:49.0879562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0879996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0880563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0881033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0881261Z 
2022-11-23T01:51:49.0881351Z Running tests...
2022-11-23T01:51:49.0881756Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0882287Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0882858Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0883382Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2619
2022-11-23T01:51:49.0883828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2620
2022-11-23T01:51:49.0884432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0884880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0885442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0885904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0886537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0886975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0887546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0888062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0888552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.0889036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.0889703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0890389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0890894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.0891366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.0891854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74355kpr
2022-11-23T01:51:49.0892387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74355kpr/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0892897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsv9op9s
2022-11-23T01:51:49.0893430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsv9op9s/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0894013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0894497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0894958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0895434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0895906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0896360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0896834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0897302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0897646Z ok (5.843s)
2022-11-23T01:51:49.0897779Z 
2022-11-23T01:51:49.0898058Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0898387Z Ran 1 test in 5.843s
2022-11-23T01:51:49.0898550Z 
2022-11-23T01:51:49.0898645Z OK
2022-11-23T01:51:49.0898778Z 
2022-11-23T01:51:49.0898882Z Generating XML reports...
2022-11-23T01:51:49.0899491Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012327.xml
2022-11-23T01:51:49.0900210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0900658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0901214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0901674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0901902Z 
2022-11-23T01:51:49.0902009Z Running tests...
2022-11-23T01:51:49.0902400Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0902929Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0903574Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0904153Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2734
2022-11-23T01:51:49.0904580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2735
2022-11-23T01:51:49.0905184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0905631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0906184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0906661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0907632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0908076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0908631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0909095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0909546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.0910050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.0910693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0911492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0912013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.0912466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.0912967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzwt22mw
2022-11-23T01:51:49.0913507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzwt22mw/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0914037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3yyudd7r
2022-11-23T01:51:49.0914556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3yyudd7r/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0915069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0915547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0915898Z ok (6.034s)
2022-11-23T01:51:49.0916027Z 
2022-11-23T01:51:49.0916302Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0916628Z Ran 1 test in 6.034s
2022-11-23T01:51:49.0916789Z 
2022-11-23T01:51:49.0916881Z OK
2022-11-23T01:51:49.0917014Z 
2022-11-23T01:51:49.0917123Z Generating XML reports...
2022-11-23T01:51:49.0917729Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012336.xml
2022-11-23T01:51:49.0918438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0918887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0919445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0919913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0920141Z 
2022-11-23T01:51:49.0920248Z Running tests...
2022-11-23T01:51:49.0920633Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0921161Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0921810Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0922383Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2849
2022-11-23T01:51:49.0922813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2850
2022-11-23T01:51:49.0923421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0923873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0924435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0924904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0925477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0925918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0926473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0926931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0927377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.0927873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.0928514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0929290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0929814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.0930272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.0930771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpamqd9_5c
2022-11-23T01:51:49.0931304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpamqd9_5c/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0931827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpta64kr7h
2022-11-23T01:51:49.0932340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpta64kr7h/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0932850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0933334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0933807Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0934268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0934612Z ok (6.652s)
2022-11-23T01:51:49.0934758Z 
2022-11-23T01:51:49.0935034Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0935344Z Ran 1 test in 6.652s
2022-11-23T01:51:49.0935508Z 
2022-11-23T01:51:49.0935602Z OK
2022-11-23T01:51:49.0935734Z 
2022-11-23T01:51:49.0935855Z Generating XML reports...
2022-11-23T01:51:49.0936445Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012344.xml
2022-11-23T01:51:49.0937157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0937609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0938184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0938689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0938930Z 
2022-11-23T01:51:49.0939040Z Running tests...
2022-11-23T01:51:49.0939447Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0939977Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0940519Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0941052Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2964
2022-11-23T01:51:49.0941498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2965
2022-11-23T01:51:49.0942092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0942545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0943122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0943588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0944146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0944586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0945152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0945596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0946112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.0946609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.0947484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0948161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0948685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.0949151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.0949649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbthp8mu1
2022-11-23T01:51:49.0950168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbthp8mu1/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0950701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4xq4gen
2022-11-23T01:51:49.0951229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4xq4gen/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0951722Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0952207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0952682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0953154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0953481Z ok (6.213s)
2022-11-23T01:51:49.0953626Z 
2022-11-23T01:51:49.0953901Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0954231Z Ran 1 test in 6.213s
2022-11-23T01:51:49.0954398Z 
2022-11-23T01:51:49.0954473Z OK
2022-11-23T01:51:49.0954607Z 
2022-11-23T01:51:49.0954730Z Generating XML reports...
2022-11-23T01:51:49.0955338Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012353.xml
2022-11-23T01:51:49.0956134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0956587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0957169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0957639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0957871Z 
2022-11-23T01:51:49.0957982Z Running tests...
2022-11-23T01:51:49.0958368Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0958890Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0959482Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0960019Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3079
2022-11-23T01:51:49.0960464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3080
2022-11-23T01:51:49.0961072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0961520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0962070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0962532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0963102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0963609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0964179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0964642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0965100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.0965588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.0966244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0966931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0967447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.0967904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.0968392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbshcf6uv
2022-11-23T01:51:49.0968928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbshcf6uv/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0969441Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8jkpctg
2022-11-23T01:51:49.0969976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8jkpctg/_remote_module_non_scriptable.py
2022-11-23T01:51:49.0970489Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0970974Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0971435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0971915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.0972259Z ok (5.735s)
2022-11-23T01:51:49.0972406Z 
2022-11-23T01:51:49.0972662Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0972993Z Ran 1 test in 5.735s
2022-11-23T01:51:49.0973155Z 
2022-11-23T01:51:49.0973251Z OK
2022-11-23T01:51:49.0973440Z 
2022-11-23T01:51:49.0973573Z Generating XML reports...
2022-11-23T01:51:49.0974168Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012402.xml
2022-11-23T01:51:49.0974883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0975326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0975884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0976362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0976590Z 
2022-11-23T01:51:49.0976698Z Running tests...
2022-11-23T01:51:49.0977102Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0977609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0978161Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0979235Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.597s)
2022-11-23T01:51:49.0979756Z 
2022-11-23T01:51:49.0980022Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0980393Z Ran 1 test in 1.597s
2022-11-23T01:51:49.0980555Z 
2022-11-23T01:51:49.0980664Z OK (skipped=1)
2022-11-23T01:51:49.0980817Z 
2022-11-23T01:51:49.0980939Z Generating XML reports...
2022-11-23T01:51:49.0981547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012410.xml
2022-11-23T01:51:49.0982247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0982694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0983268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0983719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0983946Z 
2022-11-23T01:51:49.0984051Z Running tests...
2022-11-23T01:51:49.0984451Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0984976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.0985504Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.0986024Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3228
2022-11-23T01:51:49.0986478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3229
2022-11-23T01:51:49.0987275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0987729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0988308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0988809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0989367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0989816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0990384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0990949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0991395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.0991892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.0992619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0993295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.0993825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.0994296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.0994636Z ok (4.239s)
2022-11-23T01:51:49.0994766Z 
2022-11-23T01:51:49.0995038Z ----------------------------------------------------------------------
2022-11-23T01:51:49.0995368Z Ran 1 test in 4.239s
2022-11-23T01:51:49.0995529Z 
2022-11-23T01:51:49.0995626Z OK
2022-11-23T01:51:49.0995761Z 
2022-11-23T01:51:49.0995865Z Generating XML reports...
2022-11-23T01:51:49.0996475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012415.xml
2022-11-23T01:51:49.0997185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.0997643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.0998199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.0998755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.0998986Z 
2022-11-23T01:51:49.0999093Z Running tests...
2022-11-23T01:51:49.0999479Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1000014Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1000580Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1001647Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.614s)
2022-11-23T01:51:49.1002169Z 
2022-11-23T01:51:49.1002436Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1002747Z Ran 1 test in 1.614s
2022-11-23T01:51:49.1002909Z 
2022-11-23T01:51:49.1003018Z OK (skipped=1)
2022-11-23T01:51:49.1003174Z 
2022-11-23T01:51:49.1003296Z Generating XML reports...
2022-11-23T01:51:49.1003885Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012421.xml
2022-11-23T01:51:49.1004599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1005044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1005620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1006074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1006303Z 
2022-11-23T01:51:49.1006416Z Running tests...
2022-11-23T01:51:49.1006818Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1007328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1007856Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1008416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3371
2022-11-23T01:51:49.1008878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3372
2022-11-23T01:51:49.1009470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1009921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1010496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1010950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1011533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1011971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1012542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1012985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1013437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1013933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1014589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1015258Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1015848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1016318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1016647Z ok (5.735s)
2022-11-23T01:51:49.1016794Z 
2022-11-23T01:51:49.1017069Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1017400Z Ran 1 test in 5.735s
2022-11-23T01:51:49.1017563Z 
2022-11-23T01:51:49.1017659Z OK
2022-11-23T01:51:49.1017773Z 
2022-11-23T01:51:49.1017897Z Generating XML reports...
2022-11-23T01:51:49.1018507Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012425.xml
2022-11-23T01:51:49.1019221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1019657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1020233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1020693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1020920Z 
2022-11-23T01:51:49.1021028Z Running tests...
2022-11-23T01:51:49.1021415Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1021946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1022446Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-11-23T01:51:49.1022737Z 
2022-11-23T01:51:49.1023000Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1023307Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1023466Z 
2022-11-23T01:51:49.1023574Z OK (skipped=1)
2022-11-23T01:51:49.1023732Z 
2022-11-23T01:51:49.1023854Z Generating XML reports...
2022-11-23T01:51:49.1024438Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012434.xml
2022-11-23T01:51:49.1025155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1025663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1026260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1026710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1027218Z 
2022-11-23T01:51:49.1027336Z Running tests...
2022-11-23T01:51:49.1027755Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1028267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1028736Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.1029214Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1029692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3515
2022-11-23T01:51:49.1030125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3516
2022-11-23T01:51:49.1030733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1031186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1031741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1032214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1032791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1033341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1033895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1034359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1034821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1035307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1035967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1036655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1037183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1037673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqif7_lvu
2022-11-23T01:51:49.1038210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqif7_lvu/_remote_module_non_scriptable.py
2022-11-23T01:51:49.1038718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1039214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskn_98uk
2022-11-23T01:51:49.1039729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskn_98uk/_remote_module_non_scriptable.py
2022-11-23T01:51:49.1040104Z ok (4.255s)
2022-11-23T01:51:49.1040252Z 
2022-11-23T01:51:49.1040528Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1040837Z Ran 1 test in 4.255s
2022-11-23T01:51:49.1041000Z 
2022-11-23T01:51:49.1041096Z OK
2022-11-23T01:51:49.1041230Z 
2022-11-23T01:51:49.1041355Z Generating XML reports...
2022-11-23T01:51:49.1041967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012436.xml
2022-11-23T01:51:49.1042660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1043116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1043771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1044236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1044465Z 
2022-11-23T01:51:49.1044571Z Running tests...
2022-11-23T01:51:49.1044976Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1045512Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1045975Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.1046479Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1046958Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3628
2022-11-23T01:51:49.1047385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3629
2022-11-23T01:51:49.1047994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1048444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1049019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1049471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1050043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1050567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1051141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1051586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1052047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1052554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1053192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1053880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1054396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1054896Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbngx_ka
2022-11-23T01:51:49.1055416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbngx_ka/_remote_module_non_scriptable.py
2022-11-23T01:51:49.1055919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1056415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jvn4lci
2022-11-23T01:51:49.1056946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jvn4lci/_remote_module_non_scriptable.py
2022-11-23T01:51:49.1057442Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.1057922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.1058272Z ok (4.244s)
2022-11-23T01:51:49.1058419Z 
2022-11-23T01:51:49.1058674Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1059009Z Ran 1 test in 4.244s
2022-11-23T01:51:49.1059171Z 
2022-11-23T01:51:49.1059266Z OK
2022-11-23T01:51:49.1059401Z 
2022-11-23T01:51:49.1059504Z Generating XML reports...
2022-11-23T01:51:49.1060110Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012443.xml
2022-11-23T01:51:49.1060884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1061349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1061914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1062382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1062614Z 
2022-11-23T01:51:49.1062724Z Running tests...
2022-11-23T01:51:49.1063129Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1063643Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1064137Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.1064673Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1065158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3741
2022-11-23T01:51:49.1065606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3742
2022-11-23T01:51:49.1066216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1066665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1067503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1068080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1068666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1069087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1069664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1070122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1070571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1071054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1071712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1072400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1072931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1073414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77h2gklt
2022-11-23T01:51:49.1073952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77h2gklt/_remote_module_non_scriptable.py
2022-11-23T01:51:49.1074460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1074942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo8of78zg
2022-11-23T01:51:49.1075476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo8of78zg/_remote_module_non_scriptable.py
2022-11-23T01:51:49.1075990Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.1076476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.1076811Z ok (4.366s)
2022-11-23T01:51:49.1076956Z 
2022-11-23T01:51:49.1077231Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1077563Z Ran 1 test in 4.366s
2022-11-23T01:51:49.1077725Z 
2022-11-23T01:51:49.1077801Z OK
2022-11-23T01:51:49.1077936Z 
2022-11-23T01:51:49.1078132Z Generating XML reports...
2022-11-23T01:51:49.1078760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012450.xml
2022-11-23T01:51:49.1079474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1079904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1080480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1080946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1081181Z 
2022-11-23T01:51:49.1081294Z Running tests...
2022-11-23T01:51:49.1081677Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1082204Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1082691Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T01:51:49.1083166Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1083641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3854
2022-11-23T01:51:49.1084088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3855
2022-11-23T01:51:49.1084674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1085127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1085775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1086246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1086811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1087257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1087824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1088286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1088759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1089264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1089930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1090596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1091123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1091629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgxrkbhk
2022-11-23T01:51:49.1092165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgxrkbhk/_remote_module_non_scriptable.py
2022-11-23T01:51:49.1092650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1093146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8c0_2ft2
2022-11-23T01:51:49.1093673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8c0_2ft2/_remote_module_non_scriptable.py
2022-11-23T01:51:49.1094051Z ok (4.336s)
2022-11-23T01:51:49.1094179Z 
2022-11-23T01:51:49.1094456Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1094785Z Ran 1 test in 4.336s
2022-11-23T01:51:49.1094946Z 
2022-11-23T01:51:49.1095042Z OK
2022-11-23T01:51:49.1095176Z 
2022-11-23T01:51:49.1095281Z Generating XML reports...
2022-11-23T01:51:49.1095950Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012457.xml
2022-11-23T01:51:49.1096686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1097140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1097696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1098166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1098401Z 
2022-11-23T01:51:49.1098510Z Running tests...
2022-11-23T01:51:49.1098897Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1099425Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1099919Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1100395Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3967
2022-11-23T01:51:49.1100827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3968
2022-11-23T01:51:49.1101427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1101876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1102429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1103028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1103611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1104059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1104615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1105078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1105524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1106005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1106662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1107623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1108148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1108604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1109188Z STAGE:2022-11-23 01:25:07 3967:3967 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1109764Z STAGE:2022-11-23 01:25:07 3968:3968 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1110546Z STAGE:2022-11-23 01:25:07 3968:3968 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:07 3967:3967 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1110928Z 
2022-11-23T01:51:49.1111475Z STAGE:2022-11-23 01:25:07 3968:3968 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:07 3967:3967 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1111878Z 
2022-11-23T01:51:49.1112207Z STAGE:2022-11-23 01:25:07 3967:3967 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1112773Z STAGE:2022-11-23 01:25:07 3968:3968 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1113448Z STAGE:2022-11-23 01:25:07 3968:3968 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1114030Z STAGE:2022-11-23 01:25:07 3967:3967 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1114628Z STAGE:2022-11-23 01:25:07 3968:3968 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1115217Z STAGE:2022-11-23 01:25:07 3967:3967 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1115571Z ok (4.260s)
2022-11-23T01:51:49.1115717Z 
2022-11-23T01:51:49.1115967Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1116297Z Ran 1 test in 4.261s
2022-11-23T01:51:49.1116457Z 
2022-11-23T01:51:49.1116550Z OK
2022-11-23T01:51:49.1116682Z 
2022-11-23T01:51:49.1116787Z Generating XML reports...
2022-11-23T01:51:49.1117401Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012504.xml
2022-11-23T01:51:49.1118120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1118571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1119130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1119601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1119830Z 
2022-11-23T01:51:49.1119937Z Running tests...
2022-11-23T01:51:49.1120201Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1120605Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1120866Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1121092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4080
2022-11-23T01:51:49.1121311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4081
2022-11-23T01:51:49.1121687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1121862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1122238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1122428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1122794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1122948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1123326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1123521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1123764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1124007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1124408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1124803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1125038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1125265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1125581Z STAGE:2022-11-23 01:25:14 4081:4081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1125962Z STAGE:2022-11-23 01:25:14 4080:4080 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1126718Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1126837Z   warnings.warn(
2022-11-23T01:51:49.1127563Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1127681Z   warnings.warn(
2022-11-23T01:51:49.1128016Z STAGE:2022-11-23 01:25:14 4080:4080 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1128343Z STAGE:2022-11-23 01:25:14 4081:4081 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1128688Z STAGE:2022-11-23 01:25:14 4080:4080 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1129030Z STAGE:2022-11-23 01:25:14 4081:4081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1129533Z STAGE:2022-11-23 01:25:14 4081:4081 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:25:14 4080:4080 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1129575Z 
2022-11-23T01:51:49.1129884Z STAGE:2022-11-23 01:25:14 4080:4080 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1130280Z STAGE:2022-11-23 01:25:14 4081:4081 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1130621Z STAGE:2022-11-23 01:25:14 4080:4080 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1130962Z STAGE:2022-11-23 01:25:14 4081:4081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1131289Z STAGE:2022-11-23 01:25:14 4080:4080 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1131610Z STAGE:2022-11-23 01:25:14 4081:4081 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1131936Z STAGE:2022-11-23 01:25:14 4080:4080 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1132255Z STAGE:2022-11-23 01:25:14 4081:4081 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1132599Z STAGE:2022-11-23 01:25:14 4080:4080 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1132919Z STAGE:2022-11-23 01:25:14 4081:4081 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1133022Z ok (4.232s)
2022-11-23T01:51:49.1133043Z 
2022-11-23T01:51:49.1133310Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1133423Z Ran 1 test in 4.232s
2022-11-23T01:51:49.1133443Z 
2022-11-23T01:51:49.1133535Z OK
2022-11-23T01:51:49.1133554Z 
2022-11-23T01:51:49.1133676Z Generating XML reports...
2022-11-23T01:51:49.1134130Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012510.xml
2022-11-23T01:51:49.1134500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1134659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1135046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1135237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1135257Z 
2022-11-23T01:51:49.1135363Z Running tests...
2022-11-23T01:51:49.1135680Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1136010Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1136293Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1136515Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4193
2022-11-23T01:51:49.1136733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4194
2022-11-23T01:51:49.1137085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1137268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1137644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1137834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1138199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1138371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1138745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1138933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1139161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1139402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1139866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1140261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1140495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1140724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1140965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1141205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1141603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1141987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1142320Z STAGE:2022-11-23 01:25:21 4193:4193 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1143063Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1143179Z   warnings.warn(
2022-11-23T01:51:49.1143507Z STAGE:2022-11-23 01:25:21 4194:4194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1144227Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1144344Z   warnings.warn(
2022-11-23T01:51:49.1144887Z STAGE:2022-11-23 01:25:21 4194:4194 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:21 4193:4193 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1144908Z 
2022-11-23T01:51:49.1145516Z STAGE:2022-11-23 01:25:21 4193:4193 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:21 4194:4194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1145539Z 
2022-11-23T01:51:49.1146076Z STAGE:2022-11-23 01:25:21 4194:4194 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:25:21 4193:4193 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1146095Z 
2022-11-23T01:51:49.1146430Z STAGE:2022-11-23 01:25:21 4193:4193 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1147005Z STAGE:2022-11-23 01:25:21 4194:4194 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1147364Z STAGE:2022-11-23 01:25:21 4193:4193 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1147710Z STAGE:2022-11-23 01:25:21 4194:4194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1148038Z STAGE:2022-11-23 01:25:21 4194:4194 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1148363Z STAGE:2022-11-23 01:25:21 4193:4193 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1148692Z STAGE:2022-11-23 01:25:21 4193:4193 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1149017Z STAGE:2022-11-23 01:25:21 4194:4194 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1149355Z STAGE:2022-11-23 01:25:21 4193:4193 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1149796Z STAGE:2022-11-23 01:25:21 4194:4194 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1149901Z ok (4.219s)
2022-11-23T01:51:49.1149922Z 
2022-11-23T01:51:49.1150168Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1150283Z Ran 1 test in 4.219s
2022-11-23T01:51:49.1150303Z 
2022-11-23T01:51:49.1150400Z OK
2022-11-23T01:51:49.1150421Z 
2022-11-23T01:51:49.1150546Z Generating XML reports...
2022-11-23T01:51:49.1150999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012517.xml
2022-11-23T01:51:49.1151369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1151547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1151929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1152126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1152147Z 
2022-11-23T01:51:49.1152237Z Running tests...
2022-11-23T01:51:49.1152502Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1152816Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1153089Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1153306Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4312
2022-11-23T01:51:49.1153519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4313
2022-11-23T01:51:49.1153890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1154064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1154428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1154619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1155045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1155231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1155606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1155796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1156044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1156290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1156700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1157079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1157315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1157541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1157697Z skip: Skipped due to small world size. (4.124s)
2022-11-23T01:51:49.1157717Z 
2022-11-23T01:51:49.1157984Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1158096Z Ran 1 test in 4.124s
2022-11-23T01:51:49.1158116Z 
2022-11-23T01:51:49.1158223Z OK (skipped=1)
2022-11-23T01:51:49.1158242Z 
2022-11-23T01:51:49.1158363Z Generating XML reports...
2022-11-23T01:51:49.1158796Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012524.xml
2022-11-23T01:51:49.1159245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1159425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1159805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1159997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1160017Z 
2022-11-23T01:51:49.1160126Z Running tests...
2022-11-23T01:51:49.1160392Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1160704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1160979Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1161180Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4421
2022-11-23T01:51:49.1161396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4422
2022-11-23T01:51:49.1161766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1161943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1162323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1162513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1162872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1163043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1163418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1163593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1163837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1164081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1164534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1164945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1165180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1165408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1165738Z STAGE:2022-11-23 01:25:34 4421:4421 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1166052Z STAGE:2022-11-23 01:25:34 4422:4422 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1166793Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1166911Z   warnings.warn(
2022-11-23T01:51:49.1167641Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1167753Z   warnings.warn(
2022-11-23T01:51:49.1168292Z STAGE:2022-11-23 01:25:34 4421:4421 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:34 4422:4422 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1168367Z 
2022-11-23T01:51:49.1168938Z STAGE:2022-11-23 01:25:34 4422:4422 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:34 4421:4421 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1168958Z 
2022-11-23T01:51:49.1169287Z STAGE:2022-11-23 01:25:34 4422:4422 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1169611Z STAGE:2022-11-23 01:25:34 4421:4421 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1169940Z STAGE:2022-11-23 01:25:34 4422:4422 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1170267Z STAGE:2022-11-23 01:25:34 4421:4421 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1170606Z STAGE:2022-11-23 01:25:34 4422:4422 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1170937Z STAGE:2022-11-23 01:25:34 4421:4421 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1171256Z STAGE:2022-11-23 01:25:34 4422:4422 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1171578Z STAGE:2022-11-23 01:25:34 4421:4421 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1171910Z STAGE:2022-11-23 01:25:34 4421:4421 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1172233Z STAGE:2022-11-23 01:25:34 4422:4422 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1172572Z STAGE:2022-11-23 01:25:34 4421:4421 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1172909Z STAGE:2022-11-23 01:25:34 4422:4422 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1173010Z ok (4.238s)
2022-11-23T01:51:49.1173030Z 
2022-11-23T01:51:49.1173282Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1173395Z Ran 1 test in 4.238s
2022-11-23T01:51:49.1173415Z 
2022-11-23T01:51:49.1173507Z OK
2022-11-23T01:51:49.1173526Z 
2022-11-23T01:51:49.1173648Z Generating XML reports...
2022-11-23T01:51:49.1174150Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012530.xml
2022-11-23T01:51:49.1174538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1174717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1175096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1175292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1175312Z 
2022-11-23T01:51:49.1175402Z Running tests...
2022-11-23T01:51:49.1175675Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1175986Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1176263Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1176482Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4534
2022-11-23T01:51:49.1176697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4535
2022-11-23T01:51:49.1177068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1177242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1177600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1177789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1178218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1178393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1178770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1178959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1179204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1179444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1179845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1180222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1180461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1180690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1181023Z STAGE:2022-11-23 01:25:41 4534:4534 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1181351Z STAGE:2022-11-23 01:25:41 4535:4535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1182082Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1182199Z   warnings.warn(
2022-11-23T01:51:49.1182926Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1183040Z   warnings.warn(
2022-11-23T01:51:49.1183373Z STAGE:2022-11-23 01:25:41 4534:4534 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1183744Z STAGE:2022-11-23 01:25:41 4535:4535 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1184104Z STAGE:2022-11-23 01:25:41 4534:4534 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1184446Z STAGE:2022-11-23 01:25:41 4535:4535 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1184552Z ok (4.213s)
2022-11-23T01:51:49.1184573Z 
2022-11-23T01:51:49.1184838Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1184953Z Ran 1 test in 4.213s
2022-11-23T01:51:49.1184978Z 
2022-11-23T01:51:49.1185077Z OK
2022-11-23T01:51:49.1185096Z 
2022-11-23T01:51:49.1185220Z Generating XML reports...
2022-11-23T01:51:49.1185659Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012537.xml
2022-11-23T01:51:49.1186032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1186209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1186589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1187008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1187031Z 
2022-11-23T01:51:49.1187153Z Running tests...
2022-11-23T01:51:49.1187426Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1187742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1188155Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1188357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4647
2022-11-23T01:51:49.1188603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4648
2022-11-23T01:51:49.1188990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1189171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1189550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1189742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1190106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1190284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1190639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1190826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1191078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1191320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1191726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1192120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1192350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1192688Z STAGE:2022-11-23 01:25:48 4648:4648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1192912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1193215Z STAGE:2022-11-23 01:25:48 4647:4647 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1193618Z STAGE:2022-11-23 01:25:48 4648:4648 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1193963Z STAGE:2022-11-23 01:25:48 4647:4647 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1194308Z STAGE:2022-11-23 01:25:48 4648:4648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1194646Z STAGE:2022-11-23 01:25:48 4647:4647 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1194963Z STAGE:2022-11-23 01:25:48 4648:4648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1195292Z STAGE:2022-11-23 01:25:48 4647:4647 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1195619Z STAGE:2022-11-23 01:25:48 4647:4647 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1195940Z STAGE:2022-11-23 01:25:48 4648:4648 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1196484Z STAGE:2022-11-23 01:25:48 4647:4647 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:48 4648:4648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1196524Z 
2022-11-23T01:51:49.1196609Z ok (4.255s)
2022-11-23T01:51:49.1196628Z 
2022-11-23T01:51:49.1196894Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1197005Z Ran 1 test in 4.256s
2022-11-23T01:51:49.1197025Z 
2022-11-23T01:51:49.1197116Z OK
2022-11-23T01:51:49.1197135Z 
2022-11-23T01:51:49.1197257Z Generating XML reports...
2022-11-23T01:51:49.1197776Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012544.xml
2022-11-23T01:51:49.1198147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1198322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1198687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1198883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1198902Z 
2022-11-23T01:51:49.1199012Z Running tests...
2022-11-23T01:51:49.1199273Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1199592Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1199853Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-11-23T01:51:49.1199877Z 
2022-11-23T01:51:49.1200140Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1200251Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1200270Z 
2022-11-23T01:51:49.1200375Z OK (skipped=1)
2022-11-23T01:51:49.1200395Z 
2022-11-23T01:51:49.1200498Z Generating XML reports...
2022-11-23T01:51:49.1200954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012551.xml
2022-11-23T01:51:49.1201329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1201506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1201886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1202078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1202102Z 
2022-11-23T01:51:49.1202214Z Running tests...
2022-11-23T01:51:49.1202481Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1202775Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1203103Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-11-23T01:51:49.1203124Z 
2022-11-23T01:51:49.1203397Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1203511Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1203531Z 
2022-11-23T01:51:49.1203641Z OK (skipped=1)
2022-11-23T01:51:49.1203660Z 
2022-11-23T01:51:49.1203782Z Generating XML reports...
2022-11-23T01:51:49.1204230Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012553.xml
2022-11-23T01:51:49.1204602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1204780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1205143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1205338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1205358Z 
2022-11-23T01:51:49.1205468Z Running tests...
2022-11-23T01:51:49.1205734Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1206048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1206308Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1206526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4826
2022-11-23T01:51:49.1206739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4827
2022-11-23T01:51:49.1207180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1207337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1207717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1207910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1208272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1208444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1208812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1208999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1209248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1209477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1209879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1210280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1210512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1210741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1210980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1211220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1211624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1212012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1212393Z STAGE:2022-11-23 01:25:59 4826:4826 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1212718Z STAGE:2022-11-23 01:25:59 4827:4827 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1213257Z STAGE:2022-11-23 01:25:59 4827:4827 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:25:59 4826:4826 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1213278Z 
2022-11-23T01:51:49.1213839Z STAGE:2022-11-23 01:25:59 4827:4827 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:25:59 4826:4826 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1213863Z 
2022-11-23T01:51:49.1214189Z STAGE:2022-11-23 01:25:59 4826:4826 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1214510Z STAGE:2022-11-23 01:25:59 4827:4827 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1214844Z STAGE:2022-11-23 01:25:59 4826:4826 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1215174Z STAGE:2022-11-23 01:25:59 4827:4827 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1215511Z STAGE:2022-11-23 01:25:59 4826:4826 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1215850Z STAGE:2022-11-23 01:25:59 4827:4827 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1215951Z ok (4.380s)
2022-11-23T01:51:49.1215971Z 
2022-11-23T01:51:49.1216217Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1216388Z Ran 1 test in 4.381s
2022-11-23T01:51:49.1216408Z 
2022-11-23T01:51:49.1216502Z OK
2022-11-23T01:51:49.1216521Z 
2022-11-23T01:51:49.1216647Z Generating XML reports...
2022-11-23T01:51:49.1217101Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012555.xml
2022-11-23T01:51:49.1217476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1217656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1218037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1218209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1218247Z 
2022-11-23T01:51:49.1218336Z Running tests...
2022-11-23T01:51:49.1218605Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1218924Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1219182Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1219399Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4945
2022-11-23T01:51:49.1219618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4946
2022-11-23T01:51:49.1219990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1220164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1220526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1220715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1221074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1221251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1221622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1221858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1222117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1222361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1222748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1223143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1223380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1223610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1223771Z skip: Skipped due to small world size. (4.240s)
2022-11-23T01:51:49.1223792Z 
2022-11-23T01:51:49.1224060Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1224173Z Ran 1 test in 4.241s
2022-11-23T01:51:49.1224193Z 
2022-11-23T01:51:49.1224300Z OK (skipped=1)
2022-11-23T01:51:49.1224319Z 
2022-11-23T01:51:49.1224441Z Generating XML reports...
2022-11-23T01:51:49.1224874Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012602.xml
2022-11-23T01:51:49.1225246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1225420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1225865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1226058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1226078Z 
2022-11-23T01:51:49.1226188Z Running tests...
2022-11-23T01:51:49.1226454Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1227013Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1227337Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-11-23T01:51:49.1227360Z 
2022-11-23T01:51:49.1227611Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1227726Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1227745Z 
2022-11-23T01:51:49.1227853Z OK (skipped=1)
2022-11-23T01:51:49.1227872Z 
2022-11-23T01:51:49.1228004Z Generating XML reports...
2022-11-23T01:51:49.1228456Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012609.xml
2022-11-23T01:51:49.1228823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1229001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1229385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1229575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1229594Z 
2022-11-23T01:51:49.1229684Z Running tests...
2022-11-23T01:51:49.1229948Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1230260Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1230558Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-11-23T01:51:49.1230582Z 
2022-11-23T01:51:49.1230840Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1230952Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1230972Z 
2022-11-23T01:51:49.1231078Z OK (skipped=1)
2022-11-23T01:51:49.1231098Z 
2022-11-23T01:51:49.1231321Z Generating XML reports...
2022-11-23T01:51:49.1231775Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012612.xml
2022-11-23T01:51:49.1232148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1232326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1232710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1232908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1232928Z 
2022-11-23T01:51:49.1233037Z Running tests...
2022-11-23T01:51:49.1233301Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1233612Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1233903Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-11-23T01:51:49.1233924Z 
2022-11-23T01:51:49.1234165Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1234277Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1234296Z 
2022-11-23T01:51:49.1234402Z OK (skipped=1)
2022-11-23T01:51:49.1234422Z 
2022-11-23T01:51:49.1234544Z Generating XML reports...
2022-11-23T01:51:49.1234993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012614.xml
2022-11-23T01:51:49.1235459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1235635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1236015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1236212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1236232Z 
2022-11-23T01:51:49.1236322Z Running tests...
2022-11-23T01:51:49.1236587Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1236898Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1237196Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-11-23T01:51:49.1237216Z 
2022-11-23T01:51:49.1237475Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1237592Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1237612Z 
2022-11-23T01:51:49.1237718Z OK (skipped=1)
2022-11-23T01:51:49.1237738Z 
2022-11-23T01:51:49.1237859Z Generating XML reports...
2022-11-23T01:51:49.1238311Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012616.xml
2022-11-23T01:51:49.1238667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1238842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1239219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1239407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1239428Z 
2022-11-23T01:51:49.1239535Z Running tests...
2022-11-23T01:51:49.1239798Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1240114Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1240389Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1240643Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5186
2022-11-23T01:51:49.1240868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5187
2022-11-23T01:51:49.1241243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1241416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1241793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1241985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1242352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1242522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1242898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1243071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1243316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1243559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1243957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1244353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1244646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1244878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1244983Z ok (4.222s)
2022-11-23T01:51:49.1245003Z 
2022-11-23T01:51:49.1245279Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1245375Z Ran 1 test in 4.222s
2022-11-23T01:51:49.1245394Z 
2022-11-23T01:51:49.1245488Z OK
2022-11-23T01:51:49.1245508Z 
2022-11-23T01:51:49.1245635Z Generating XML reports...
2022-11-23T01:51:49.1246084Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012619.xml
2022-11-23T01:51:49.1246452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1246628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1247013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1247202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1247222Z 
2022-11-23T01:51:49.1247311Z Running tests...
2022-11-23T01:51:49.1247583Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1247897Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1248166Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1248381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5295
2022-11-23T01:51:49.1248595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5296
2022-11-23T01:51:49.1248968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1249147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1249523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1249694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1250125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1250310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1250687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1250874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1251123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1251375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1251775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1252155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1252387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1252613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1252852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1253091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1253489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1253942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1254185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:51:49.1254427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:51:49.1254806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.1255198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.1255440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T01:51:49.1255673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T01:51:49.1256065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T01:51:49.1256454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T01:51:49.1256557Z ok (4.347s)
2022-11-23T01:51:49.1256577Z 
2022-11-23T01:51:49.1256844Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1256956Z Ran 1 test in 4.347s
2022-11-23T01:51:49.1256976Z 
2022-11-23T01:51:49.1257050Z OK
2022-11-23T01:51:49.1257087Z 
2022-11-23T01:51:49.1257191Z Generating XML reports...
2022-11-23T01:51:49.1257643Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012626.xml
2022-11-23T01:51:49.1258011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1258187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1258573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1258764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1258784Z 
2022-11-23T01:51:49.1258892Z Running tests...
2022-11-23T01:51:49.1259256Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1259568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1259835Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-11-23T01:51:49.1259855Z 
2022-11-23T01:51:49.1260111Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1260221Z Ran 1 test in 0.003s
2022-11-23T01:51:49.1260241Z 
2022-11-23T01:51:49.1260347Z OK (skipped=1)
2022-11-23T01:51:49.1260367Z 
2022-11-23T01:51:49.1260493Z Generating XML reports...
2022-11-23T01:51:49.1260942Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012632.xml
2022-11-23T01:51:49.1261312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1261491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1261851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1262038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1262058Z 
2022-11-23T01:51:49.1262168Z Running tests...
2022-11-23T01:51:49.1262428Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1262739Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1263021Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1263300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5461
2022-11-23T01:51:49.1263521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5462
2022-11-23T01:51:49.1263883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1264063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1264443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1264633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1264995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1265167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1265543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1265730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1265975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1266205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1266607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1267249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1267484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1267716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1267964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1268209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1268695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1269106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1269418Z STAGE:2022-11-23 01:26:39 5461:5461 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1270153Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1270272Z   warnings.warn(
2022-11-23T01:51:49.1270598Z STAGE:2022-11-23 01:26:39 5462:5462 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1271319Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1271428Z   warnings.warn(
2022-11-23T01:51:49.1271758Z STAGE:2022-11-23 01:26:39 5462:5462 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1272104Z STAGE:2022-11-23 01:26:39 5462:5462 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1272426Z STAGE:2022-11-23 01:26:39 5461:5461 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1272767Z STAGE:2022-11-23 01:26:39 5461:5461 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1273163Z STAGE:2022-11-23 01:26:39 5462:5462 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1273488Z STAGE:2022-11-23 01:26:39 5461:5461 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1273819Z STAGE:2022-11-23 01:26:39 5462:5462 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1274157Z STAGE:2022-11-23 01:26:39 5462:5462 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1274487Z STAGE:2022-11-23 01:26:39 5461:5461 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1274824Z STAGE:2022-11-23 01:26:39 5461:5461 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1274922Z ok (4.369s)
2022-11-23T01:51:49.1274943Z 
2022-11-23T01:51:49.1275203Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1275300Z Ran 1 test in 4.370s
2022-11-23T01:51:49.1275335Z 
2022-11-23T01:51:49.1275410Z OK
2022-11-23T01:51:49.1275428Z 
2022-11-23T01:51:49.1275548Z Generating XML reports...
2022-11-23T01:51:49.1276002Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012635.xml
2022-11-23T01:51:49.1276371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1276540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1276919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1277107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1277126Z 
2022-11-23T01:51:49.1277233Z Running tests...
2022-11-23T01:51:49.1277478Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1277795Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1278074Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1278288Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5580
2022-11-23T01:51:49.1278552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5581
2022-11-23T01:51:49.1278939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1279111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1279482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1279653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1280010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1280178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1280541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1280721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1280955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1281191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1281590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1281981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1282193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1282478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1282713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1282954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1283351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1283737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1284058Z STAGE:2022-11-23 01:26:46 5580:5580 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1284372Z STAGE:2022-11-23 01:26:46 5581:5581 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1285105Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1285213Z   warnings.warn(
2022-11-23T01:51:49.1285931Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1286039Z   warnings.warn(
2022-11-23T01:51:49.1286364Z STAGE:2022-11-23 01:26:46 5580:5580 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1286686Z STAGE:2022-11-23 01:26:46 5581:5581 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1287023Z STAGE:2022-11-23 01:26:46 5580:5580 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1287365Z STAGE:2022-11-23 01:26:46 5581:5581 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1287930Z STAGE:2022-11-23 01:26:46 5580:5580 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:26:46 5581:5581 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1287953Z 
2022-11-23T01:51:49.1288497Z STAGE:2022-11-23 01:26:46 5580:5580 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:26:46 5581:5581 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1288517Z 
2022-11-23T01:51:49.1288891Z STAGE:2022-11-23 01:26:46 5580:5580 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1289223Z STAGE:2022-11-23 01:26:46 5581:5581 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1289312Z ok (4.237s)
2022-11-23T01:51:49.1289343Z 
2022-11-23T01:51:49.1289588Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1289695Z Ran 1 test in 4.238s
2022-11-23T01:51:49.1289714Z 
2022-11-23T01:51:49.1289798Z OK
2022-11-23T01:51:49.1289817Z 
2022-11-23T01:51:49.1289934Z Generating XML reports...
2022-11-23T01:51:49.1290383Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012642.xml
2022-11-23T01:51:49.1290749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1290917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1291286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1291459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1291540Z 
2022-11-23T01:51:49.1291648Z Running tests...
2022-11-23T01:51:49.1291903Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1292206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1292494Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1292705Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5699
2022-11-23T01:51:49.1292911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5700
2022-11-23T01:51:49.1293276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1293431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1293800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1293987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1294344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1294511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1294877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1295057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1295294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1295529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1295913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1296303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1296523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1296743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1297028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1297275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1297677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1298062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1298387Z STAGE:2022-11-23 01:26:52 5700:5700 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1298698Z STAGE:2022-11-23 01:26:52 5699:5699 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1299433Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1299543Z   warnings.warn(
2022-11-23T01:51:49.1300262Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1300367Z   warnings.warn(
2022-11-23T01:51:49.1300694Z STAGE:2022-11-23 01:26:52 5700:5700 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1301079Z STAGE:2022-11-23 01:26:52 5699:5699 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1301411Z STAGE:2022-11-23 01:26:52 5700:5700 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1301741Z STAGE:2022-11-23 01:26:52 5699:5699 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1302257Z STAGE:2022-11-23 01:26:52 5700:5700 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:26:52 5699:5699 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1302278Z 
2022-11-23T01:51:49.1302600Z STAGE:2022-11-23 01:26:52 5699:5699 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1302904Z STAGE:2022-11-23 01:26:52 5700:5700 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1303232Z STAGE:2022-11-23 01:26:52 5699:5699 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1303564Z STAGE:2022-11-23 01:26:52 5700:5700 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1303658Z ok (4.322s)
2022-11-23T01:51:49.1303678Z 
2022-11-23T01:51:49.1303931Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1304033Z Ran 1 test in 4.323s
2022-11-23T01:51:49.1304053Z 
2022-11-23T01:51:49.1304136Z OK
2022-11-23T01:51:49.1304156Z 
2022-11-23T01:51:49.1304273Z Generating XML reports...
2022-11-23T01:51:49.1304725Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012649.xml
2022-11-23T01:51:49.1305092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1305263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1305636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1305813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1305847Z 
2022-11-23T01:51:49.1305937Z Running tests...
2022-11-23T01:51:49.1306196Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1306562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1307046Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1307268Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5818
2022-11-23T01:51:49.1307477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5819
2022-11-23T01:51:49.1307857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1308028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1308395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1308584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1308937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1309110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1309480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1309659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1309895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1310131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1310616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1311003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1311226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1311448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1311677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1311913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1312307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1312698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1313029Z STAGE:2022-11-23 01:26:59 5818:5818 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1313768Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1313866Z   warnings.warn(
2022-11-23T01:51:49.1314198Z STAGE:2022-11-23 01:26:59 5819:5819 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1314919Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1315035Z   warnings.warn(
2022-11-23T01:51:49.1315579Z STAGE:2022-11-23 01:26:59 5819:5819 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:26:59 5818:5818 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1315600Z 
2022-11-23T01:51:49.1316226Z STAGE:2022-11-23 01:26:59 5818:5818 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:26:59 5819:5819 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1316250Z 
2022-11-23T01:51:49.1316585Z STAGE:2022-11-23 01:26:59 5818:5818 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1316900Z STAGE:2022-11-23 01:26:59 5819:5819 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1317220Z STAGE:2022-11-23 01:26:59 5819:5819 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1317538Z STAGE:2022-11-23 01:26:59 5818:5818 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1317873Z STAGE:2022-11-23 01:26:59 5819:5819 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1318196Z STAGE:2022-11-23 01:26:59 5818:5818 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1318289Z ok (4.214s)
2022-11-23T01:51:49.1318308Z 
2022-11-23T01:51:49.1318567Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1318670Z Ran 1 test in 4.214s
2022-11-23T01:51:49.1318689Z 
2022-11-23T01:51:49.1318771Z OK
2022-11-23T01:51:49.1318790Z 
2022-11-23T01:51:49.1318904Z Generating XML reports...
2022-11-23T01:51:49.1319351Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012655.xml
2022-11-23T01:51:49.1319717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1319875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1320336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1320520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1320540Z 
2022-11-23T01:51:49.1320637Z Running tests...
2022-11-23T01:51:49.1320897Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1321206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1321477Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1321685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5937
2022-11-23T01:51:49.1321890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5938
2022-11-23T01:51:49.1322245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1322414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1322787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1322975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1323335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1323504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1323870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1324055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1324282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1324523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1324915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1325355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1325590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1325810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1325961Z skip: Skipped due to small world size. (4.149s)
2022-11-23T01:51:49.1325981Z 
2022-11-23T01:51:49.1326240Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1326348Z Ran 1 test in 4.150s
2022-11-23T01:51:49.1326368Z 
2022-11-23T01:51:49.1326456Z OK (skipped=1)
2022-11-23T01:51:49.1326489Z 
2022-11-23T01:51:49.1326595Z Generating XML reports...
2022-11-23T01:51:49.1327038Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012702.xml
2022-11-23T01:51:49.1327400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1327569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1327940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1328121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1328141Z 
2022-11-23T01:51:49.1328246Z Running tests...
2022-11-23T01:51:49.1328502Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1328799Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1329128Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1329334Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6046
2022-11-23T01:51:49.1329540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6047
2022-11-23T01:51:49.1329915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1330082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1330451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1330633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1330978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1331143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1331510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1331688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1331928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1332164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1332556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1332938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1333157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1333368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1333524Z skip: Skipped due to small world size. (4.354s)
2022-11-23T01:51:49.1333544Z 
2022-11-23T01:51:49.1333800Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1333904Z Ran 1 test in 4.354s
2022-11-23T01:51:49.1333923Z 
2022-11-23T01:51:49.1334025Z OK (skipped=1)
2022-11-23T01:51:49.1334096Z 
2022-11-23T01:51:49.1334222Z Generating XML reports...
2022-11-23T01:51:49.1334671Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012709.xml
2022-11-23T01:51:49.1335032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1335202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1335562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1335749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1335769Z 
2022-11-23T01:51:49.1335869Z Running tests...
2022-11-23T01:51:49.1336125Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1336428Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1336707Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1336916Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6155
2022-11-23T01:51:49.1337122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6156
2022-11-23T01:51:49.1337472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1337642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1338079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1338260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1338612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1338777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1339138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1339315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1339552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1339778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1340167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1340556Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1340778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1341000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1341148Z skip: Skipped due to small world size. (4.254s)
2022-11-23T01:51:49.1341169Z 
2022-11-23T01:51:49.1341425Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1341528Z Ran 1 test in 4.254s
2022-11-23T01:51:49.1341547Z 
2022-11-23T01:51:49.1341647Z OK (skipped=1)
2022-11-23T01:51:49.1341667Z 
2022-11-23T01:51:49.1341771Z Generating XML reports...
2022-11-23T01:51:49.1342214Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012716.xml
2022-11-23T01:51:49.1342579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1342746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1343170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1343369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1343388Z 
2022-11-23T01:51:49.1343490Z Running tests...
2022-11-23T01:51:49.1343755Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1344048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1344319Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1344532Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6264
2022-11-23T01:51:49.1344751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6265
2022-11-23T01:51:49.1345117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1345290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1345668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1345857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1346219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1346376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1346745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1347282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1347526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1347769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1348183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1348574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1348796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1349016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1349156Z skip: Skipped due to small world size. (4.157s)
2022-11-23T01:51:49.1349180Z 
2022-11-23T01:51:49.1349441Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1349544Z Ran 1 test in 4.157s
2022-11-23T01:51:49.1349564Z 
2022-11-23T01:51:49.1349659Z OK (skipped=1)
2022-11-23T01:51:49.1349678Z 
2022-11-23T01:51:49.1349793Z Generating XML reports...
2022-11-23T01:51:49.1350238Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012723.xml
2022-11-23T01:51:49.1350599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1350764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1351125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1351309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1351328Z 
2022-11-23T01:51:49.1351432Z Running tests...
2022-11-23T01:51:49.1351688Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1351993Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1352256Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1352537Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6373
2022-11-23T01:51:49.1352757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6374
2022-11-23T01:51:49.1353130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1353287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1353656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1353838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1354197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1354364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1354729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1354908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1355148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1355373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1355764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1356147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1356425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1356754Z STAGE:2022-11-23 01:27:33 6374:6374 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1357482Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1357587Z   warnings.warn(
2022-11-23T01:51:49.1357808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1358130Z STAGE:2022-11-23 01:27:33 6373:6373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1358849Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1358949Z   warnings.warn(
2022-11-23T01:51:49.1359483Z STAGE:2022-11-23 01:27:33 6374:6374 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:27:33 6373:6373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1359504Z 
2022-11-23T01:51:49.1360063Z STAGE:2022-11-23 01:27:33 6373:6373 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:27:33 6374:6374 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1360084Z 
2022-11-23T01:51:49.1360401Z STAGE:2022-11-23 01:27:33 6374:6374 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1360714Z STAGE:2022-11-23 01:27:33 6373:6373 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1361046Z STAGE:2022-11-23 01:27:33 6373:6373 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1361361Z STAGE:2022-11-23 01:27:33 6374:6374 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1361747Z STAGE:2022-11-23 01:27:33 6373:6373 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1362098Z STAGE:2022-11-23 01:27:33 6374:6374 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1362197Z ok (4.340s)
2022-11-23T01:51:49.1362217Z 
2022-11-23T01:51:49.1362462Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1362567Z Ran 1 test in 4.340s
2022-11-23T01:51:49.1362586Z 
2022-11-23T01:51:49.1362676Z OK
2022-11-23T01:51:49.1362696Z 
2022-11-23T01:51:49.1362816Z Generating XML reports...
2022-11-23T01:51:49.1363260Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012729.xml
2022-11-23T01:51:49.1363631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1363806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1364179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1364367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1364386Z 
2022-11-23T01:51:49.1364477Z Running tests...
2022-11-23T01:51:49.1364738Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1365041Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1365333Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1365604Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6486
2022-11-23T01:51:49.1365815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6487
2022-11-23T01:51:49.1366188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1366359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1366720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1366908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1367259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1367426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1367789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1367980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1368225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1368465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1368862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1369242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1369466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1370195Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1370309Z   warnings.warn(
2022-11-23T01:51:49.1370527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1371308Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1371425Z   warnings.warn(
2022-11-23T01:51:49.1371520Z ok (4.359s)
2022-11-23T01:51:49.1371540Z 
2022-11-23T01:51:49.1371803Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1371912Z Ran 1 test in 4.359s
2022-11-23T01:51:49.1371932Z 
2022-11-23T01:51:49.1372006Z OK
2022-11-23T01:51:49.1372025Z 
2022-11-23T01:51:49.1372142Z Generating XML reports...
2022-11-23T01:51:49.1372597Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012736.xml
2022-11-23T01:51:49.1372964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1373132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1373508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1373691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1373711Z 
2022-11-23T01:51:49.1373813Z Running tests...
2022-11-23T01:51:49.1374062Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1374374Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1374641Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1374911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6595
2022-11-23T01:51:49.1375123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6596
2022-11-23T01:51:49.1375497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1375666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1376038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1376225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1376577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1376738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1377108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1377295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1377531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1377771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1378171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1378557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1378768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1378994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1379327Z STAGE:2022-11-23 01:27:47 6596:6596 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1379648Z STAGE:2022-11-23 01:27:47 6595:6595 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1380449Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1380571Z   warnings.warn(
2022-11-23T01:51:49.1381312Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1381418Z   warnings.warn(
2022-11-23T01:51:49.1381750Z STAGE:2022-11-23 01:27:47 6596:6596 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1382076Z STAGE:2022-11-23 01:27:47 6595:6595 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1382622Z STAGE:2022-11-23 01:27:47 6595:6595 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:27:47 6596:6596 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1382655Z 
2022-11-23T01:51:49.1383159Z STAGE:2022-11-23 01:27:47 6596:6596 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:27:47 6595:6595 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1383194Z 
2022-11-23T01:51:49.1383499Z STAGE:2022-11-23 01:27:47 6596:6596 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1383828Z STAGE:2022-11-23 01:27:47 6595:6595 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1384241Z STAGE:2022-11-23 01:27:47 6596:6596 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1384572Z STAGE:2022-11-23 01:27:47 6595:6595 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1384669Z ok (4.232s)
2022-11-23T01:51:49.1384689Z 
2022-11-23T01:51:49.1384952Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1385059Z Ran 1 test in 4.232s
2022-11-23T01:51:49.1385078Z 
2022-11-23T01:51:49.1385166Z OK
2022-11-23T01:51:49.1385185Z 
2022-11-23T01:51:49.1385292Z Generating XML reports...
2022-11-23T01:51:49.1385744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012743.xml
2022-11-23T01:51:49.1386108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1386281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1386662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1387076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1387099Z 
2022-11-23T01:51:49.1387210Z Running tests...
2022-11-23T01:51:49.1387483Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1387794Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1388054Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1388266Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6708
2022-11-23T01:51:49.1388480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6709
2022-11-23T01:51:49.1388888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1389061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1389436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1389624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1390061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1390230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1390612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1390797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1391039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1391278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1391681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1392073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1392299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1392524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1393039Z STAGE:2022-11-23 01:27:54 6708:6708 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:27:54 6709:6709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1393071Z 
2022-11-23T01:51:49.1393788Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1393973Z   warnings.warn(
2022-11-23T01:51:49.1394714Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1394824Z   warnings.warn(
2022-11-23T01:51:49.1395152Z STAGE:2022-11-23 01:27:54 6708:6708 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1395469Z STAGE:2022-11-23 01:27:54 6709:6709 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1396026Z STAGE:2022-11-23 01:27:54 6708:6708 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:27:54 6709:6709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1396050Z 
2022-11-23T01:51:49.1396369Z STAGE:2022-11-23 01:27:54 6708:6708 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1396689Z STAGE:2022-11-23 01:27:54 6709:6709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1397011Z STAGE:2022-11-23 01:27:54 6708:6708 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1397325Z STAGE:2022-11-23 01:27:54 6709:6709 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1397657Z STAGE:2022-11-23 01:27:54 6708:6708 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1397991Z STAGE:2022-11-23 01:27:54 6709:6709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1398087Z ok (4.220s)
2022-11-23T01:51:49.1398107Z 
2022-11-23T01:51:49.1398362Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1398475Z Ran 1 test in 4.220s
2022-11-23T01:51:49.1398495Z 
2022-11-23T01:51:49.1398585Z OK
2022-11-23T01:51:49.1398604Z 
2022-11-23T01:51:49.1398726Z Generating XML reports...
2022-11-23T01:51:49.1399175Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012750.xml
2022-11-23T01:51:49.1399583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1399764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1400147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1400331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1400351Z 
2022-11-23T01:51:49.1400453Z Running tests...
2022-11-23T01:51:49.1400711Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1401021Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1401289Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1401489Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6821
2022-11-23T01:51:49.1401706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6822
2022-11-23T01:51:49.1402071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1402240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1402612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1402794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1403149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1403433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1403804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1403977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1404220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1404460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1404859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1405248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1405473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1405805Z STAGE:2022-11-23 01:28:00 6822:6822 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1406533Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1406646Z   warnings.warn(
2022-11-23T01:51:49.1406857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1407190Z STAGE:2022-11-23 01:28:00 6821:6821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1407914Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T01:51:49.1408026Z   warnings.warn(
2022-11-23T01:51:49.1408353Z STAGE:2022-11-23 01:28:00 6822:6822 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1408673Z STAGE:2022-11-23 01:28:00 6821:6821 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1409057Z STAGE:2022-11-23 01:28:00 6822:6822 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1409410Z STAGE:2022-11-23 01:28:00 6821:6821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1409730Z STAGE:2022-11-23 01:28:00 6822:6822 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1410029Z STAGE:2022-11-23 01:28:00 6821:6821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1410346Z STAGE:2022-11-23 01:28:00 6822:6822 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1410677Z STAGE:2022-11-23 01:28:00 6821:6821 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1411012Z STAGE:2022-11-23 01:28:00 6822:6822 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1411340Z STAGE:2022-11-23 01:28:00 6821:6821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1411441Z ok (4.339s)
2022-11-23T01:51:49.1411461Z 
2022-11-23T01:51:49.1411720Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1411830Z Ran 1 test in 4.339s
2022-11-23T01:51:49.1411849Z 
2022-11-23T01:51:49.1411932Z OK
2022-11-23T01:51:49.1411951Z 
2022-11-23T01:51:49.1412057Z Generating XML reports...
2022-11-23T01:51:49.1412502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012757.xml
2022-11-23T01:51:49.1412871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1413100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1413478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1413671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1413691Z 
2022-11-23T01:51:49.1413798Z Running tests...
2022-11-23T01:51:49.1414053Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1414347Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1414631Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1414845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6934
2022-11-23T01:51:49.1415050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6935
2022-11-23T01:51:49.1415425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1415596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1415970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1416155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1416515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1416669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1417037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1417219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1417468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1417708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1418151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1418560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1418789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1419014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1419098Z ok (4.245s)
2022-11-23T01:51:49.1419118Z 
2022-11-23T01:51:49.1419377Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1419487Z Ran 1 test in 4.246s
2022-11-23T01:51:49.1419510Z 
2022-11-23T01:51:49.1419601Z OK
2022-11-23T01:51:49.1419620Z 
2022-11-23T01:51:49.1419741Z Generating XML reports...
2022-11-23T01:51:49.1420193Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012803.xml
2022-11-23T01:51:49.1420566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1420740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1421105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1421295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1421314Z 
2022-11-23T01:51:49.1421415Z Running tests...
2022-11-23T01:51:49.1421672Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1421980Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1422304Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1422518Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7043
2022-11-23T01:51:49.1422736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7044
2022-11-23T01:51:49.1423105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1423262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1423633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1423819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1424176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1424345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1424710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1424899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1425145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1425370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1425766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1426158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1426383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1426606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1427091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1427424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1427843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1428227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1428550Z STAGE:2022-11-23 01:28:14 7043:7043 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1428859Z STAGE:2022-11-23 01:28:14 7044:7044 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1429188Z STAGE:2022-11-23 01:28:14 7043:7043 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1429509Z STAGE:2022-11-23 01:28:14 7044:7044 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1429847Z STAGE:2022-11-23 01:28:14 7043:7043 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1430187Z STAGE:2022-11-23 01:28:14 7044:7044 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1430501Z STAGE:2022-11-23 01:28:14 7044:7044 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1430818Z STAGE:2022-11-23 01:28:14 7043:7043 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1431350Z STAGE:2022-11-23 01:28:14 7043:7043 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:28:14 7044:7044 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1431372Z 
2022-11-23T01:51:49.1431919Z STAGE:2022-11-23 01:28:14 7043:7043 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:28:14 7044:7044 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1432009Z 
2022-11-23T01:51:49.1432117Z ok (4.335s)
2022-11-23T01:51:49.1432137Z 
2022-11-23T01:51:49.1432388Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1432504Z Ran 1 test in 4.335s
2022-11-23T01:51:49.1432523Z 
2022-11-23T01:51:49.1432611Z OK
2022-11-23T01:51:49.1432630Z 
2022-11-23T01:51:49.1432748Z Generating XML reports...
2022-11-23T01:51:49.1433200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012810.xml
2022-11-23T01:51:49.1433569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1433738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1434118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1434290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1434323Z 
2022-11-23T01:51:49.1434413Z Running tests...
2022-11-23T01:51:49.1434672Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1434987Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1435247Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1435460Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7162
2022-11-23T01:51:49.1435673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7163
2022-11-23T01:51:49.1436034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1436210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1436574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1436761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1437168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1437349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1437721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1437901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1438144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1438386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1438772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1439162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1439392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1439613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1439845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1440081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1440477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1440928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1441246Z STAGE:2022-11-23 01:28:21 7162:7162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1441568Z STAGE:2022-11-23 01:28:21 7163:7163 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1441887Z STAGE:2022-11-23 01:28:21 7162:7162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1442212Z STAGE:2022-11-23 01:28:21 7163:7163 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1442765Z STAGE:2022-11-23 01:28:21 7162:7162 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:28:21 7163:7163 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1442785Z 
2022-11-23T01:51:49.1443102Z STAGE:2022-11-23 01:28:21 7162:7162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1443429Z STAGE:2022-11-23 01:28:21 7163:7163 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1443748Z STAGE:2022-11-23 01:28:21 7162:7162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1444065Z STAGE:2022-11-23 01:28:21 7163:7163 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1444405Z STAGE:2022-11-23 01:28:21 7162:7162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1444743Z STAGE:2022-11-23 01:28:21 7163:7163 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1444827Z ok (4.268s)
2022-11-23T01:51:49.1444847Z 
2022-11-23T01:51:49.1445107Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1445215Z Ran 1 test in 4.268s
2022-11-23T01:51:49.1445235Z 
2022-11-23T01:51:49.1445326Z OK
2022-11-23T01:51:49.1445345Z 
2022-11-23T01:51:49.1445460Z Generating XML reports...
2022-11-23T01:51:49.1445916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012817.xml
2022-11-23T01:51:49.1446285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1446528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1446926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1447101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1447120Z 
2022-11-23T01:51:49.1447227Z Running tests...
2022-11-23T01:51:49.1447485Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1447794Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1448067Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1448288Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7281
2022-11-23T01:51:49.1448503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7282
2022-11-23T01:51:49.1448871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1449027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1449402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1449589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1449944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1450111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1450550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1450739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1450981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1451232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1451616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1452009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1452236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1452459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1452701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1452935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1453332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1453724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1454053Z STAGE:2022-11-23 01:28:28 7282:7282 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1454363Z STAGE:2022-11-23 01:28:28 7281:7281 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1454895Z STAGE:2022-11-23 01:28:28 7281:7281 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:28:28 7282:7282 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1454921Z 
2022-11-23T01:51:49.1455479Z STAGE:2022-11-23 01:28:28 7281:7281 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:28:28 7282:7282 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1455499Z 
2022-11-23T01:51:49.1455873Z STAGE:2022-11-23 01:28:28 7282:7282 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1456203Z STAGE:2022-11-23 01:28:28 7281:7281 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1456532Z STAGE:2022-11-23 01:28:28 7281:7281 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1456858Z STAGE:2022-11-23 01:28:28 7282:7282 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1457191Z STAGE:2022-11-23 01:28:28 7281:7281 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1457532Z STAGE:2022-11-23 01:28:28 7282:7282 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1457631Z ok (4.366s)
2022-11-23T01:51:49.1457651Z 
2022-11-23T01:51:49.1457898Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1458006Z Ran 1 test in 4.367s
2022-11-23T01:51:49.1458026Z 
2022-11-23T01:51:49.1458111Z OK
2022-11-23T01:51:49.1458134Z 
2022-11-23T01:51:49.1458255Z Generating XML reports...
2022-11-23T01:51:49.1458708Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012824.xml
2022-11-23T01:51:49.1459069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1459241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1459622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1459872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1459892Z 
2022-11-23T01:51:49.1459982Z Running tests...
2022-11-23T01:51:49.1460248Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1460562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1460834Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1461048Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7400
2022-11-23T01:51:49.1461258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7401
2022-11-23T01:51:49.1461619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1461790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1462148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1462341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1462694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1462866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1463234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1463417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1463661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1463902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1464297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1464683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1464909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1465186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1465432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1465670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1466070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1466458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1467114Z STAGE:2022-11-23 01:28:35 7401:7401 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1467475Z STAGE:2022-11-23 01:28:35 7400:7400 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1467789Z STAGE:2022-11-23 01:28:35 7400:7400 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1468115Z STAGE:2022-11-23 01:28:35 7401:7401 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1468459Z STAGE:2022-11-23 01:28:35 7400:7400 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1468788Z STAGE:2022-11-23 01:28:35 7401:7401 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1469299Z STAGE:2022-11-23 01:28:35 7401:7401 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:28:35 7400:7400 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1469408Z 
2022-11-23T01:51:49.1469749Z STAGE:2022-11-23 01:28:35 7400:7400 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1470066Z STAGE:2022-11-23 01:28:35 7401:7401 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1470408Z STAGE:2022-11-23 01:28:35 7400:7400 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1470742Z STAGE:2022-11-23 01:28:35 7401:7401 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1470826Z ok (4.338s)
2022-11-23T01:51:49.1470862Z 
2022-11-23T01:51:49.1471109Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1471214Z Ran 1 test in 4.339s
2022-11-23T01:51:49.1471234Z 
2022-11-23T01:51:49.1471322Z OK
2022-11-23T01:51:49.1471341Z 
2022-11-23T01:51:49.1471461Z Generating XML reports...
2022-11-23T01:51:49.1471906Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012831.xml
2022-11-23T01:51:49.1472279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1472453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1472832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1473006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1473025Z 
2022-11-23T01:51:49.1473131Z Running tests...
2022-11-23T01:51:49.1473395Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1473702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1473962Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1474177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7519
2022-11-23T01:51:49.1474393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7520
2022-11-23T01:51:49.1474762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1474919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1475366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1475566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1475933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1476104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1476473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1476665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1476911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1477150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1477536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1477930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1478156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1478377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1478532Z skip: Skipped due to small world size. (4.248s)
2022-11-23T01:51:49.1478604Z 
2022-11-23T01:51:49.1478878Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1478989Z Ran 1 test in 4.248s
2022-11-23T01:51:49.1479009Z 
2022-11-23T01:51:49.1479114Z OK (skipped=1)
2022-11-23T01:51:49.1479133Z 
2022-11-23T01:51:49.1479254Z Generating XML reports...
2022-11-23T01:51:49.1479692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012838.xml
2022-11-23T01:51:49.1480062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1480229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1480604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1480791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1480812Z 
2022-11-23T01:51:49.1480915Z Running tests...
2022-11-23T01:51:49.1481180Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1481489Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1481735Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1481956Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7628
2022-11-23T01:51:49.1482168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7629
2022-11-23T01:51:49.1482537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1482708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1483081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1483268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1483635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1483803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1484207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1484402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1484646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1484889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1485287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1485679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1485910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1486133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1486271Z skip: Skipped due to small world size. (4.224s)
2022-11-23T01:51:49.1486312Z 
2022-11-23T01:51:49.1486558Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1486669Z Ran 1 test in 4.224s
2022-11-23T01:51:49.1486690Z 
2022-11-23T01:51:49.1486794Z OK (skipped=1)
2022-11-23T01:51:49.1486813Z 
2022-11-23T01:51:49.1486932Z Generating XML reports...
2022-11-23T01:51:49.1487380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012844.xml
2022-11-23T01:51:49.1487747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1488033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1488420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1488592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1488656Z 
2022-11-23T01:51:49.1488753Z Running tests...
2022-11-23T01:51:49.1489019Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1489332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1489602Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1489819Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7737
2022-11-23T01:51:49.1490034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7738
2022-11-23T01:51:49.1490408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1490582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1490942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1491134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1491497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1491666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1492042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1492229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1492472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1492715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1493098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1493546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1493784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1494010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1494168Z skip: Skipped due to small world size. (4.150s)
2022-11-23T01:51:49.1494189Z 
2022-11-23T01:51:49.1494453Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1494562Z Ran 1 test in 4.150s
2022-11-23T01:51:49.1494582Z 
2022-11-23T01:51:49.1494691Z OK (skipped=1)
2022-11-23T01:51:49.1494710Z 
2022-11-23T01:51:49.1494832Z Generating XML reports...
2022-11-23T01:51:49.1495266Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012851.xml
2022-11-23T01:51:49.1495640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1495813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1496193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1496380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1496400Z 
2022-11-23T01:51:49.1496507Z Running tests...
2022-11-23T01:51:49.1496768Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1497076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1497404Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1497602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7846
2022-11-23T01:51:49.1497816Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7847
2022-11-23T01:51:49.1498191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1498364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1498747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1498935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1499296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1499473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1499826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1500011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1500258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1500500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1500899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1501292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1501520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1501750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1501905Z skip: Skipped due to small world size. (4.354s)
2022-11-23T01:51:49.1501925Z 
2022-11-23T01:51:49.1502170Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1502281Z Ran 1 test in 4.354s
2022-11-23T01:51:49.1502300Z 
2022-11-23T01:51:49.1502454Z OK (skipped=1)
2022-11-23T01:51:49.1502475Z 
2022-11-23T01:51:49.1502603Z Generating XML reports...
2022-11-23T01:51:49.1503056Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012858.xml
2022-11-23T01:51:49.1503422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1503595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1503969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1504166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1504185Z 
2022-11-23T01:51:49.1504276Z Running tests...
2022-11-23T01:51:49.1504537Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1504849Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1505099Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1505314Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7955
2022-11-23T01:51:49.1505526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7956
2022-11-23T01:51:49.1505895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1506068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1506494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1506685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1507326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1507502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1507879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1508067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1508311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1508554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1508957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1509336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1509568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1509796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1510126Z STAGE:2022-11-23 01:29:09 7955:7955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1510449Z STAGE:2022-11-23 01:29:09 7956:7956 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1510779Z STAGE:2022-11-23 01:29:09 7955:7955 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1511101Z STAGE:2022-11-23 01:29:09 7956:7956 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1511447Z STAGE:2022-11-23 01:29:09 7955:7955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1511766Z STAGE:2022-11-23 01:29:09 7956:7956 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1512179Z STAGE:2022-11-23 01:29:09 7955:7955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1512527Z STAGE:2022-11-23 01:29:09 7956:7956 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1512855Z STAGE:2022-11-23 01:29:09 7955:7955 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1513173Z STAGE:2022-11-23 01:29:09 7956:7956 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1513512Z STAGE:2022-11-23 01:29:09 7955:7955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1513846Z STAGE:2022-11-23 01:29:09 7956:7956 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1513951Z ok (4.261s)
2022-11-23T01:51:49.1513973Z 
2022-11-23T01:51:49.1514238Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1514329Z Ran 1 test in 4.261s
2022-11-23T01:51:49.1514349Z 
2022-11-23T01:51:49.1514440Z OK
2022-11-23T01:51:49.1514459Z 
2022-11-23T01:51:49.1514580Z Generating XML reports...
2022-11-23T01:51:49.1515032Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012905.xml
2022-11-23T01:51:49.1515406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1515578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1515951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1516139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1516227Z 
2022-11-23T01:51:49.1516344Z Running tests...
2022-11-23T01:51:49.1516595Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1516904Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1517159Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1517375Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8068
2022-11-23T01:51:49.1517588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8069
2022-11-23T01:51:49.1517959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1518130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1518505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1518681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1519046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1519218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1519591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1519780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1520024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1520264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1520664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1521064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1521274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1521500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1521883Z STAGE:2022-11-23 01:29:16 8068:8068 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1522226Z STAGE:2022-11-23 01:29:16 8069:8069 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1522553Z STAGE:2022-11-23 01:29:16 8068:8068 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1522877Z STAGE:2022-11-23 01:29:16 8069:8069 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1523217Z STAGE:2022-11-23 01:29:16 8068:8068 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1523562Z STAGE:2022-11-23 01:29:16 8069:8069 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1523883Z STAGE:2022-11-23 01:29:16 8068:8068 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1524190Z STAGE:2022-11-23 01:29:16 8069:8069 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1524728Z STAGE:2022-11-23 01:29:16 8069:8069 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:29:16 8068:8068 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1524750Z 
2022-11-23T01:51:49.1525312Z STAGE:2022-11-23 01:29:16 8069:8069 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:29:16 8068:8068 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1525332Z 
2022-11-23T01:51:49.1525431Z ok (4.261s)
2022-11-23T01:51:49.1525450Z 
2022-11-23T01:51:49.1525777Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1525891Z Ran 1 test in 4.262s
2022-11-23T01:51:49.1525911Z 
2022-11-23T01:51:49.1526001Z OK
2022-11-23T01:51:49.1526021Z 
2022-11-23T01:51:49.1526142Z Generating XML reports...
2022-11-23T01:51:49.1526595Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012912.xml
2022-11-23T01:51:49.1526967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1527127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1527510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1527697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1527717Z 
2022-11-23T01:51:49.1527822Z Running tests...
2022-11-23T01:51:49.1528087Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1528394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1528655Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1528873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8181
2022-11-23T01:51:49.1529073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8182
2022-11-23T01:51:49.1529444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1529616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1529989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1530179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1530545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1530718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1531083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1531324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1531561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1531806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1532206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1532601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1532836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1533062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1533393Z STAGE:2022-11-23 01:29:24 8181:8181 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1533721Z STAGE:2022-11-23 01:29:24 8182:8182 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1534492Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:51:49.1534589Z   warnings.warn(
2022-11-23T01:51:49.1535348Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:51:49.1535520Z   warnings.warn(
2022-11-23T01:51:49.1536070Z STAGE:2022-11-23 01:29:24 8182:8182 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:29:24 8181:8181 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1536091Z 
2022-11-23T01:51:49.1536652Z STAGE:2022-11-23 01:29:24 8181:8181 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:29:24 8182:8182 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1536673Z 
2022-11-23T01:51:49.1537187Z STAGE:2022-11-23 01:29:24 8182:8182 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:29:24 8181:8181 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1537211Z 
2022-11-23T01:51:49.1537546Z STAGE:2022-11-23 01:29:24 8181:8181 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1537872Z STAGE:2022-11-23 01:29:24 8182:8182 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1538212Z STAGE:2022-11-23 01:29:24 8181:8181 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1538556Z STAGE:2022-11-23 01:29:24 8182:8182 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1538655Z ok (6.047s)
2022-11-23T01:51:49.1538675Z 
2022-11-23T01:51:49.1538937Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1539031Z Ran 1 test in 6.048s
2022-11-23T01:51:49.1539051Z 
2022-11-23T01:51:49.1539140Z OK
2022-11-23T01:51:49.1539159Z 
2022-11-23T01:51:49.1539279Z Generating XML reports...
2022-11-23T01:51:49.1539730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012918.xml
2022-11-23T01:51:49.1540104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1540276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1540706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1540906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1540926Z 
2022-11-23T01:51:49.1541015Z Running tests...
2022-11-23T01:51:49.1541280Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1541590Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1541864Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1542076Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8296
2022-11-23T01:51:49.1542294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8297
2022-11-23T01:51:49.1542667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1542839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1543221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1543395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1543759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1543928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1544295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1544539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1544784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1545027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1545434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1545814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1546043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1546269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1546597Z STAGE:2022-11-23 01:29:33 8297:8297 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1547619Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:51:49.1547735Z   warnings.warn(
2022-11-23T01:51:49.1548072Z STAGE:2022-11-23 01:29:33 8296:8296 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1548829Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:51:49.1548940Z   warnings.warn(
2022-11-23T01:51:49.1549276Z STAGE:2022-11-23 01:29:33 8296:8296 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1549594Z STAGE:2022-11-23 01:29:33 8297:8297 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1549936Z STAGE:2022-11-23 01:29:33 8296:8296 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1550276Z STAGE:2022-11-23 01:29:33 8297:8297 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1550727Z STAGE:2022-11-23 01:29:33 8296:8296 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1551071Z STAGE:2022-11-23 01:29:33 8297:8297 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1551397Z STAGE:2022-11-23 01:29:33 8296:8296 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1551724Z STAGE:2022-11-23 01:29:33 8297:8297 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1552063Z STAGE:2022-11-23 01:29:33 8296:8296 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1552403Z STAGE:2022-11-23 01:29:33 8297:8297 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1552487Z ok (6.163s)
2022-11-23T01:51:49.1552509Z 
2022-11-23T01:51:49.1552771Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1552882Z Ran 1 test in 6.163s
2022-11-23T01:51:49.1552901Z 
2022-11-23T01:51:49.1552995Z OK
2022-11-23T01:51:49.1553014Z 
2022-11-23T01:51:49.1553137Z Generating XML reports...
2022-11-23T01:51:49.1553588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012927.xml
2022-11-23T01:51:49.1553956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1554130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1554490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1554757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1554776Z 
2022-11-23T01:51:49.1554884Z Running tests...
2022-11-23T01:51:49.1555151Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1555466Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1555727Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1555944Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8411
2022-11-23T01:51:49.1556160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8412
2022-11-23T01:51:49.1556529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1556686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1557069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1557258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1557622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1557798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1558169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1558355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1558600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1558826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1559225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1559624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1559854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1560237Z STAGE:2022-11-23 01:29:40 8412:8412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1560470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1560798Z STAGE:2022-11-23 01:29:40 8411:8411 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1561129Z STAGE:2022-11-23 01:29:40 8411:8411 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1561450Z STAGE:2022-11-23 01:29:40 8412:8412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1561779Z STAGE:2022-11-23 01:29:40 8411:8411 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1562122Z STAGE:2022-11-23 01:29:40 8412:8412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1562439Z STAGE:2022-11-23 01:29:40 8411:8411 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1562763Z STAGE:2022-11-23 01:29:40 8412:8412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1563092Z STAGE:2022-11-23 01:29:40 8411:8411 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1563411Z STAGE:2022-11-23 01:29:40 8412:8412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1563749Z STAGE:2022-11-23 01:29:40 8411:8411 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1564078Z STAGE:2022-11-23 01:29:40 8412:8412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1564246Z ok (4.228s)
2022-11-23T01:51:49.1564266Z 
2022-11-23T01:51:49.1564515Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1564627Z Ran 1 test in 4.229s
2022-11-23T01:51:49.1564647Z 
2022-11-23T01:51:49.1564737Z OK
2022-11-23T01:51:49.1564756Z 
2022-11-23T01:51:49.1564880Z Generating XML reports...
2022-11-23T01:51:49.1565335Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012936.xml
2022-11-23T01:51:49.1565704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1565879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1566254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1566427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1566467Z 
2022-11-23T01:51:49.1566558Z Running tests...
2022-11-23T01:51:49.1566821Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1567132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1567401Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1567615Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8524
2022-11-23T01:51:49.1567829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8525
2022-11-23T01:51:49.1568201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1568374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1568730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1568925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1569288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1569459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1569875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1570073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1570318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1570561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1570964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1571350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1571581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1571807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1571911Z ok (5.725s)
2022-11-23T01:51:49.1571932Z 
2022-11-23T01:51:49.1572196Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1572306Z Ran 1 test in 5.725s
2022-11-23T01:51:49.1572326Z 
2022-11-23T01:51:49.1572416Z OK
2022-11-23T01:51:49.1572435Z 
2022-11-23T01:51:49.1572556Z Generating XML reports...
2022-11-23T01:51:49.1572989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012942.xml
2022-11-23T01:51:49.1573359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1573596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1573979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1574167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1574187Z 
2022-11-23T01:51:49.1574297Z Running tests...
2022-11-23T01:51:49.1574557Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1574864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1575115Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1575314Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8635
2022-11-23T01:51:49.1575528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8636
2022-11-23T01:51:49.1575902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1576074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1576450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1576641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1577003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1577177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1577531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1577717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1577958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1578205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1578607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1579065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1579306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1579640Z STAGE:2022-11-23 01:29:55 8636:8636 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1579865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1580167Z STAGE:2022-11-23 01:29:55 8635:8635 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1580706Z STAGE:2022-11-23 01:29:55 8636:8636 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:29:55 8635:8635 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1580731Z 
2022-11-23T01:51:49.1581295Z STAGE:2022-11-23 01:29:55 8636:8636 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:29:55 8635:8635 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1581315Z 
2022-11-23T01:51:49.1581642Z STAGE:2022-11-23 01:29:55 8636:8636 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1581958Z STAGE:2022-11-23 01:29:55 8635:8635 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1582284Z STAGE:2022-11-23 01:29:55 8636:8636 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1582602Z STAGE:2022-11-23 01:29:55 8635:8635 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1583005Z STAGE:2022-11-23 01:29:55 8636:8636 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1583340Z STAGE:2022-11-23 01:29:55 8635:8635 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1583440Z ok (4.256s)
2022-11-23T01:51:49.1583460Z 
2022-11-23T01:51:49.1583711Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1583825Z Ran 1 test in 4.257s
2022-11-23T01:51:49.1583844Z 
2022-11-23T01:51:49.1583936Z OK
2022-11-23T01:51:49.1583955Z 
2022-11-23T01:51:49.1584077Z Generating XML reports...
2022-11-23T01:51:49.1584531Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012951.xml
2022-11-23T01:51:49.1584901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1585072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1585457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1585648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1585668Z 
2022-11-23T01:51:49.1585758Z Running tests...
2022-11-23T01:51:49.1586020Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1586334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1586594Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1587044Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8748
2022-11-23T01:51:49.1587274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8749
2022-11-23T01:51:49.1587652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1587833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1588192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1588381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1588856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1589045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1589420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1589609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1589853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1590098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1590504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1590884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1591115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1591343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1591673Z STAGE:2022-11-23 01:30:01 8748:8748 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1591995Z STAGE:2022-11-23 01:30:01 8749:8749 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1592322Z STAGE:2022-11-23 01:30:01 8748:8748 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1592732Z STAGE:2022-11-23 01:30:01 8749:8749 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1593072Z STAGE:2022-11-23 01:30:01 8748:8748 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1593411Z STAGE:2022-11-23 01:30:01 8749:8749 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1593718Z STAGE:2022-11-23 01:30:01 8748:8748 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1594040Z STAGE:2022-11-23 01:30:01 8749:8749 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1594365Z STAGE:2022-11-23 01:30:01 8748:8748 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1594688Z STAGE:2022-11-23 01:30:01 8749:8749 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1595027Z STAGE:2022-11-23 01:30:01 8748:8748 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1595366Z STAGE:2022-11-23 01:30:01 8749:8749 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1595466Z ok (4.239s)
2022-11-23T01:51:49.1595487Z 
2022-11-23T01:51:49.1595749Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1595842Z Ran 1 test in 4.239s
2022-11-23T01:51:49.1595877Z 
2022-11-23T01:51:49.1595950Z OK
2022-11-23T01:51:49.1595973Z 
2022-11-23T01:51:49.1596094Z Generating XML reports...
2022-11-23T01:51:49.1596546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012958.xml
2022-11-23T01:51:49.1596920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1597095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1597474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1597670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1597690Z 
2022-11-23T01:51:49.1597795Z Running tests...
2022-11-23T01:51:49.1598043Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1598356Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1598673Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1598901Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8861
2022-11-23T01:51:49.1599116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8862
2022-11-23T01:51:49.1599488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1599660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1600041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1600214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1600574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1600748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1601119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1601304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1601545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1601788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1602185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1602646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1602857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1603193Z STAGE:2022-11-23 01:30:08 8862:8862 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1603421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1603749Z STAGE:2022-11-23 01:30:08 8861:8861 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1604077Z STAGE:2022-11-23 01:30:08 8861:8861 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1604401Z STAGE:2022-11-23 01:30:08 8862:8862 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1604746Z STAGE:2022-11-23 01:30:08 8861:8861 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1605077Z STAGE:2022-11-23 01:30:08 8862:8862 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1605391Z STAGE:2022-11-23 01:30:08 8861:8861 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1605697Z STAGE:2022-11-23 01:30:08 8862:8862 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1606025Z STAGE:2022-11-23 01:30:08 8861:8861 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1606344Z STAGE:2022-11-23 01:30:08 8862:8862 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1606682Z STAGE:2022-11-23 01:30:08 8861:8861 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1607016Z STAGE:2022-11-23 01:30:08 8862:8862 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1607120Z ok (4.251s)
2022-11-23T01:51:49.1607141Z 
2022-11-23T01:51:49.1607403Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1607513Z Ran 1 test in 4.251s
2022-11-23T01:51:49.1607533Z 
2022-11-23T01:51:49.1607606Z OK
2022-11-23T01:51:49.1607641Z 
2022-11-23T01:51:49.1607746Z Generating XML reports...
2022-11-23T01:51:49.1608249Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013004.xml
2022-11-23T01:51:49.1608635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1608811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1609184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1609372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1609397Z 
2022-11-23T01:51:49.1609505Z Running tests...
2022-11-23T01:51:49.1609770Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1610062Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1610321Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1610539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8974
2022-11-23T01:51:49.1610751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8975
2022-11-23T01:51:49.1611120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1611292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1611669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1611919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1612288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1612443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1612820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1613010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1613256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1613500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1613898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1614296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1614523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1614734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1615069Z STAGE:2022-11-23 01:30:17 8975:8975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1615395Z STAGE:2022-11-23 01:30:17 8974:8974 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1615723Z STAGE:2022-11-23 01:30:17 8974:8974 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1616045Z STAGE:2022-11-23 01:30:17 8975:8975 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1616383Z STAGE:2022-11-23 01:30:17 8974:8974 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1616730Z STAGE:2022-11-23 01:30:17 8975:8975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1617048Z STAGE:2022-11-23 01:30:17 8975:8975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1617368Z STAGE:2022-11-23 01:30:17 8974:8974 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1617932Z STAGE:2022-11-23 01:30:18 8975:8975 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:30:18 8974:8974 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1617974Z 
2022-11-23T01:51:49.1618308Z STAGE:2022-11-23 01:30:18 8974:8974 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1618644Z STAGE:2022-11-23 01:30:18 8975:8975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1618963Z STAGE:2022-11-23 01:30:18 8975:8975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1619288Z STAGE:2022-11-23 01:30:18 8974:8974 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1619812Z STAGE:2022-11-23 01:30:18 8975:8975 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:30:18 8974:8974 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1619833Z 
2022-11-23T01:51:49.1620172Z STAGE:2022-11-23 01:30:18 8974:8974 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1620506Z STAGE:2022-11-23 01:30:18 8975:8975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1620607Z ok (7.449s)
2022-11-23T01:51:49.1620627Z 
2022-11-23T01:51:49.1620889Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1620982Z Ran 1 test in 7.449s
2022-11-23T01:51:49.1621017Z 
2022-11-23T01:51:49.1621090Z OK
2022-11-23T01:51:49.1621110Z 
2022-11-23T01:51:49.1621230Z Generating XML reports...
2022-11-23T01:51:49.1621746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013011.xml
2022-11-23T01:51:49.1622115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1622291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1622675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1622867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1622886Z 
2022-11-23T01:51:49.1622993Z Running tests...
2022-11-23T01:51:49.1623237Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1623547Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1623814Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1624034Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9090
2022-11-23T01:51:49.1624247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9091
2022-11-23T01:51:49.1624620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1624795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1625173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1625344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1625707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1625875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1626256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1626441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1626685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1627308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1627747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1628142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1628355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1628586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1628922Z STAGE:2022-11-23 01:30:27 9090:9090 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1629245Z STAGE:2022-11-23 01:30:27 9091:9091 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1629572Z STAGE:2022-11-23 01:30:27 9090:9090 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1629908Z STAGE:2022-11-23 01:30:27 9091:9091 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1630250Z STAGE:2022-11-23 01:30:27 9090:9090 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1630590Z STAGE:2022-11-23 01:30:27 9091:9091 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1630906Z STAGE:2022-11-23 01:30:27 9090:9090 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1631209Z STAGE:2022-11-23 01:30:27 9091:9091 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1631825Z STAGE:2022-11-23 01:30:28 9091:9091 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:30:28 9090:9090 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1631847Z 
2022-11-23T01:51:49.1632408Z STAGE:2022-11-23 01:30:28 9090:9090 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:30:28 9091:9091 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1632428Z 
2022-11-23T01:51:49.1632951Z STAGE:2022-11-23 01:30:28 9090:9090 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:30:28 9091:9091 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1632971Z 
2022-11-23T01:51:49.1633298Z STAGE:2022-11-23 01:30:28 9090:9090 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1633617Z STAGE:2022-11-23 01:30:28 9091:9091 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1633956Z STAGE:2022-11-23 01:30:28 9090:9090 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1634292Z STAGE:2022-11-23 01:30:28 9091:9091 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1634393Z ok (7.454s)
2022-11-23T01:51:49.1634412Z 
2022-11-23T01:51:49.1634677Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1634787Z Ran 1 test in 7.454s
2022-11-23T01:51:49.1634806Z 
2022-11-23T01:51:49.1634879Z OK
2022-11-23T01:51:49.1634898Z 
2022-11-23T01:51:49.1635022Z Generating XML reports...
2022-11-23T01:51:49.1635471Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013021.xml
2022-11-23T01:51:49.1635844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1636018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1636404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1636596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1636616Z 
2022-11-23T01:51:49.1636722Z Running tests...
2022-11-23T01:51:49.1637018Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1637348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1637623Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1637837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9206
2022-11-23T01:51:49.1638050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9207
2022-11-23T01:51:49.1638416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1638595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1638973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1639160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1639508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1639680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1640052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1640238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1640479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1640783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1641189Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1641587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1641819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1642030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1642358Z STAGE:2022-11-23 01:30:37 9207:9207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1642681Z STAGE:2022-11-23 01:30:37 9206:9206 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1643008Z STAGE:2022-11-23 01:30:37 9206:9206 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1643334Z STAGE:2022-11-23 01:30:37 9207:9207 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1643672Z STAGE:2022-11-23 01:30:37 9206:9206 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1644008Z STAGE:2022-11-23 01:30:37 9207:9207 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1644533Z STAGE:2022-11-23 01:30:37 9206:9206 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:30:37 9207:9207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1644554Z 
2022-11-23T01:51:49.1645090Z STAGE:2022-11-23 01:30:38 9206:9206 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:30:38 9207:9207 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1645110Z 
2022-11-23T01:51:49.1645663Z STAGE:2022-11-23 01:30:38 9207:9207 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:30:38 9206:9206 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1645688Z 
2022-11-23T01:51:49.1646010Z STAGE:2022-11-23 01:30:38 9207:9207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1646382Z STAGE:2022-11-23 01:30:38 9206:9206 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1646943Z STAGE:2022-11-23 01:30:38 9207:9207 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:30:38 9206:9206 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1646963Z 
2022-11-23T01:51:49.1647304Z STAGE:2022-11-23 01:30:38 9207:9207 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1647641Z STAGE:2022-11-23 01:30:38 9206:9206 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1647741Z ok (7.517s)
2022-11-23T01:51:49.1647761Z 
2022-11-23T01:51:49.1648030Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1648144Z Ran 1 test in 7.517s
2022-11-23T01:51:49.1648164Z 
2022-11-23T01:51:49.1648255Z OK
2022-11-23T01:51:49.1648273Z 
2022-11-23T01:51:49.1648396Z Generating XML reports...
2022-11-23T01:51:49.1648833Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013031.xml
2022-11-23T01:51:49.1649211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1649388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1649768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1649956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1649976Z 
2022-11-23T01:51:49.1650082Z Running tests...
2022-11-23T01:51:49.1650409Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1650722Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1650967Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:51:49.1650988Z 
2022-11-23T01:51:49.1651233Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1651346Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1651365Z 
2022-11-23T01:51:49.1651470Z OK (skipped=1)
2022-11-23T01:51:49.1651489Z 
2022-11-23T01:51:49.1651611Z Generating XML reports...
2022-11-23T01:51:49.1652056Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013041.xml
2022-11-23T01:51:49.1652420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1652593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1652971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1653160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1653180Z 
2022-11-23T01:51:49.1653269Z Running tests...
2022-11-23T01:51:49.1653535Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1653846Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1654101Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:51:49.1654121Z 
2022-11-23T01:51:49.1654377Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1654486Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1654505Z 
2022-11-23T01:51:49.1654610Z OK (skipped=1)
2022-11-23T01:51:49.1654628Z 
2022-11-23T01:51:49.1654752Z Generating XML reports...
2022-11-23T01:51:49.1655183Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013044.xml
2022-11-23T01:51:49.1655556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1655782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1656176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1656364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1656383Z 
2022-11-23T01:51:49.1656490Z Running tests...
2022-11-23T01:51:49.1656815Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1657175Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1657477Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T01:51:49.1657504Z 
2022-11-23T01:51:49.1657748Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1657899Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1657920Z 
2022-11-23T01:51:49.1658223Z OK (skipped=1)
2022-11-23T01:51:49.1658243Z 
2022-11-23T01:51:49.1658413Z Generating XML reports...
2022-11-23T01:51:49.1658922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013046.xml
2022-11-23T01:51:49.1659282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1659498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1659916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1660223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1660243Z 
2022-11-23T01:51:49.1660386Z Running tests...
2022-11-23T01:51:49.1660688Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1661102Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1661421Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T01:51:49.1661442Z 
2022-11-23T01:51:49.1661748Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1661844Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1661863Z 
2022-11-23T01:51:49.1662008Z OK (skipped=1)
2022-11-23T01:51:49.1662028Z 
2022-11-23T01:51:49.1662185Z Generating XML reports...
2022-11-23T01:51:49.1662674Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013048.xml
2022-11-23T01:51:49.1663087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1663371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1663836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1664072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1664092Z 
2022-11-23T01:51:49.1664239Z Running tests...
2022-11-23T01:51:49.1664489Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1664836Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1665128Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:51:49.1665148Z 
2022-11-23T01:51:49.1665447Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1665612Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1665633Z 
2022-11-23T01:51:49.1665774Z OK (skipped=1)
2022-11-23T01:51:49.1665793Z 
2022-11-23T01:51:49.1665990Z Generating XML reports...
2022-11-23T01:51:49.1666488Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013051.xml
2022-11-23T01:51:49.1667183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1667433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1667871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1668102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1668122Z 
2022-11-23T01:51:49.1668278Z Running tests...
2022-11-23T01:51:49.1668630Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1668987Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1669353Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T01:51:49.1669375Z 
2022-11-23T01:51:49.1669688Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1669785Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1669805Z 
2022-11-23T01:51:49.1669951Z OK (skipped=1)
2022-11-23T01:51:49.1669970Z 
2022-11-23T01:51:49.1670126Z Generating XML reports...
2022-11-23T01:51:49.1670629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013053.xml
2022-11-23T01:51:49.1671037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1671251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1671753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1672028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1672049Z 
2022-11-23T01:51:49.1672195Z Running tests...
2022-11-23T01:51:49.1672449Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1672811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1673104Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T01:51:49.1673125Z 
2022-11-23T01:51:49.1673422Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1673629Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1673649Z 
2022-11-23T01:51:49.1673795Z OK (skipped=1)
2022-11-23T01:51:49.1673815Z 
2022-11-23T01:51:49.1673974Z Generating XML reports...
2022-11-23T01:51:49.1674509Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013055.xml
2022-11-23T01:51:49.1674938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1675101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1675521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1675746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1675767Z 
2022-11-23T01:51:49.1675908Z Running tests...
2022-11-23T01:51:49.1676209Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1676562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1676879Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:51:49.1676904Z 
2022-11-23T01:51:49.1677252Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1677349Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1677421Z 
2022-11-23T01:51:49.1677513Z OK (skipped=1)
2022-11-23T01:51:49.1677532Z 
2022-11-23T01:51:49.1677747Z Generating XML reports...
2022-11-23T01:51:49.1678253Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013058.xml
2022-11-23T01:51:49.1678712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1678927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1679349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1679588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1679612Z 
2022-11-23T01:51:49.1679801Z Running tests...
2022-11-23T01:51:49.1680055Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1680405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1680732Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:51:49.1680753Z 
2022-11-23T01:51:49.1681052Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1681197Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1681217Z 
2022-11-23T01:51:49.1681361Z OK (skipped=1)
2022-11-23T01:51:49.1681381Z 
2022-11-23T01:51:49.1681548Z Generating XML reports...
2022-11-23T01:51:49.1682038Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013100.xml
2022-11-23T01:51:49.1682481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1682706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1683128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1683358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1683378Z 
2022-11-23T01:51:49.1683576Z Running tests...
2022-11-23T01:51:49.1683888Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1684243Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1684577Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:51:49.1684598Z 
2022-11-23T01:51:49.1684897Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1685094Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1685115Z 
2022-11-23T01:51:49.1685208Z OK (skipped=1)
2022-11-23T01:51:49.1685227Z 
2022-11-23T01:51:49.1685388Z Generating XML reports...
2022-11-23T01:51:49.1685875Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013103.xml
2022-11-23T01:51:49.1686296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1686509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1686928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1687155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1687176Z 
2022-11-23T01:51:49.1687317Z Running tests...
2022-11-23T01:51:49.1687563Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1687966Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1688296Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:51:49.1688317Z 
2022-11-23T01:51:49.1688777Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1688943Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1688963Z 
2022-11-23T01:51:49.1689107Z OK (skipped=1)
2022-11-23T01:51:49.1689127Z 
2022-11-23T01:51:49.1689285Z Generating XML reports...
2022-11-23T01:51:49.1689781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013105.xml
2022-11-23T01:51:49.1690189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1690350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1690826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1691064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1691085Z 
2022-11-23T01:51:49.1691230Z Running tests...
2022-11-23T01:51:49.1691534Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1691884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1692228Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:51:49.1692250Z 
2022-11-23T01:51:49.1692550Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1692670Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1724906Z 
2022-11-23T01:51:49.1725074Z OK (skipped=1)
2022-11-23T01:51:49.1725253Z 
2022-11-23T01:51:49.1725396Z Generating XML reports...
2022-11-23T01:51:49.1725897Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013107.xml
2022-11-23T01:51:49.1726268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1726440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1726815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1726996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1727017Z 
2022-11-23T01:51:49.1727115Z Running tests...
2022-11-23T01:51:49.1727366Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1727668Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1727964Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:51:49.1727984Z 
2022-11-23T01:51:49.1728238Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1728339Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1728359Z 
2022-11-23T01:51:49.1728459Z OK (skipped=1)
2022-11-23T01:51:49.1728479Z 
2022-11-23T01:51:49.1728593Z Generating XML reports...
2022-11-23T01:51:49.1729034Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013110.xml
2022-11-23T01:51:49.1729393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1729554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1729924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1730111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1730131Z 
2022-11-23T01:51:49.1730227Z Running tests...
2022-11-23T01:51:49.1730477Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1730871Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1731181Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:51:49.1731202Z 
2022-11-23T01:51:49.1731456Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1731556Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1731576Z 
2022-11-23T01:51:49.1731664Z OK (skipped=1)
2022-11-23T01:51:49.1731683Z 
2022-11-23T01:51:49.1731795Z Generating XML reports...
2022-11-23T01:51:49.1732234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013112.xml
2022-11-23T01:51:49.1732600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1732766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1733140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1733321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1733342Z 
2022-11-23T01:51:49.1733438Z Running tests...
2022-11-23T01:51:49.1733692Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1733989Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1734269Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:51:49.1734347Z 
2022-11-23T01:51:49.1734610Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1734709Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1734728Z 
2022-11-23T01:51:49.1734825Z OK (skipped=1)
2022-11-23T01:51:49.1734843Z 
2022-11-23T01:51:49.1734954Z Generating XML reports...
2022-11-23T01:51:49.1735401Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013115.xml
2022-11-23T01:51:49.1735762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1735920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1736289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1736467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1736487Z 
2022-11-23T01:51:49.1736587Z Running tests...
2022-11-23T01:51:49.1736837Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1737138Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1737429Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:51:49.1737449Z 
2022-11-23T01:51:49.1737699Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1737799Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1737819Z 
2022-11-23T01:51:49.1737908Z OK (skipped=1)
2022-11-23T01:51:49.1737933Z 
2022-11-23T01:51:49.1738039Z Generating XML reports...
2022-11-23T01:51:49.1738472Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013117.xml
2022-11-23T01:51:49.1738832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1739002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1739375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1739554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1739630Z 
2022-11-23T01:51:49.1739737Z Running tests...
2022-11-23T01:51:49.1740000Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1740294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1740580Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:51:49.1740601Z 
2022-11-23T01:51:49.1740860Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1740969Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1740994Z 
2022-11-23T01:51:49.1741098Z OK (skipped=1)
2022-11-23T01:51:49.1741118Z 
2022-11-23T01:51:49.1741240Z Generating XML reports...
2022-11-23T01:51:49.1741688Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013119.xml
2022-11-23T01:51:49.1742060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1742238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1742601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1742792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1742812Z 
2022-11-23T01:51:49.1742917Z Running tests...
2022-11-23T01:51:49.1743176Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1743485Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1743845Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:51:49.1743865Z 
2022-11-23T01:51:49.1744122Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1744235Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1744255Z 
2022-11-23T01:51:49.1744361Z OK (skipped=1)
2022-11-23T01:51:49.1744381Z 
2022-11-23T01:51:49.1744487Z Generating XML reports...
2022-11-23T01:51:49.1744935Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013122.xml
2022-11-23T01:51:49.1745299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1745474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1745848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1746042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1746063Z 
2022-11-23T01:51:49.1746170Z Running tests...
2022-11-23T01:51:49.1746430Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1746728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1747253Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:51:49.1747275Z 
2022-11-23T01:51:49.1747544Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1747654Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1747674Z 
2022-11-23T01:51:49.1747779Z OK (skipped=1)
2022-11-23T01:51:49.1747798Z 
2022-11-23T01:51:49.1747918Z Generating XML reports...
2022-11-23T01:51:49.1748370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013124.xml
2022-11-23T01:51:49.1748736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1748912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1749404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1749611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1749631Z 
2022-11-23T01:51:49.1749738Z Running tests...
2022-11-23T01:51:49.1750007Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1750317Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1750622Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:51:49.1750648Z 
2022-11-23T01:51:49.1750906Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1751017Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1751037Z 
2022-11-23T01:51:49.1751143Z OK (skipped=1)
2022-11-23T01:51:49.1751162Z 
2022-11-23T01:51:49.1751271Z Generating XML reports...
2022-11-23T01:51:49.1751721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013127.xml
2022-11-23T01:51:49.1752087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1752262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1752641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1752830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1752920Z 
2022-11-23T01:51:49.1753035Z Running tests...
2022-11-23T01:51:49.1753302Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1753612Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1753897Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:51:49.1753917Z 
2022-11-23T01:51:49.1754176Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1754286Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1754306Z 
2022-11-23T01:51:49.1754411Z OK (skipped=1)
2022-11-23T01:51:49.1754431Z 
2022-11-23T01:51:49.1754549Z Generating XML reports...
2022-11-23T01:51:49.1754997Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013129.xml
2022-11-23T01:51:49.1755372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1755545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1755924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1756101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1756120Z 
2022-11-23T01:51:49.1756228Z Running tests...
2022-11-23T01:51:49.1756489Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1756800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1757108Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:51:49.1757129Z 
2022-11-23T01:51:49.1757385Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1757500Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1757520Z 
2022-11-23T01:51:49.1757625Z OK (skipped=1)
2022-11-23T01:51:49.1757645Z 
2022-11-23T01:51:49.1757765Z Generating XML reports...
2022-11-23T01:51:49.1758252Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013131.xml
2022-11-23T01:51:49.1758638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1758813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1759191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1759381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1759401Z 
2022-11-23T01:51:49.1759508Z Running tests...
2022-11-23T01:51:49.1759767Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1760084Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1760359Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T01:51:49.1760396Z 
2022-11-23T01:51:49.1760641Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1760752Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1760771Z 
2022-11-23T01:51:49.1760875Z OK (skipped=1)
2022-11-23T01:51:49.1760894Z 
2022-11-23T01:51:49.1761015Z Generating XML reports...
2022-11-23T01:51:49.1761458Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013134.xml
2022-11-23T01:51:49.1761823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1761996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1762440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1762612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1762648Z 
2022-11-23T01:51:49.1762737Z Running tests...
2022-11-23T01:51:49.1762999Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1763310Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1763610Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T01:51:49.1763630Z 
2022-11-23T01:51:49.1763886Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1763997Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1764017Z 
2022-11-23T01:51:49.1764122Z OK (skipped=1)
2022-11-23T01:51:49.1764146Z 
2022-11-23T01:51:49.1764270Z Generating XML reports...
2022-11-23T01:51:49.1764703Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013136.xml
2022-11-23T01:51:49.1765073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1765251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1765631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1765821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1765840Z 
2022-11-23T01:51:49.1765950Z Running tests...
2022-11-23T01:51:49.1766209Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1766518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1766770Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1766990Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10114
2022-11-23T01:51:49.1767208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10115
2022-11-23T01:51:49.1767628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1767812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1768194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1768385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1768750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1768929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1769289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1769480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1769726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1769970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1770369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1770763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1770990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1771285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1771527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1771750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1772153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1772548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1772650Z ok (6.868s)
2022-11-23T01:51:49.1772670Z 
2022-11-23T01:51:49.1772929Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1773041Z Ran 1 test in 6.868s
2022-11-23T01:51:49.1773062Z 
2022-11-23T01:51:49.1773154Z OK
2022-11-23T01:51:49.1773173Z 
2022-11-23T01:51:49.1773295Z Generating XML reports...
2022-11-23T01:51:49.1773739Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013139.xml
2022-11-23T01:51:49.1774114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1774289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1774669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1774862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1774882Z 
2022-11-23T01:51:49.1774988Z Running tests...
2022-11-23T01:51:49.1775251Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1775562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1775822Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1776026Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10235
2022-11-23T01:51:49.1776240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10236
2022-11-23T01:51:49.1776664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1776847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1777223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1777412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1777777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1777951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1778332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1778504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1778748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1778995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1779400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1779797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1780027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1780254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1780460Z skip: Need at least 3 CUDA devices (4.321s)
2022-11-23T01:51:49.1780480Z 
2022-11-23T01:51:49.1780731Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1780842Z Ran 1 test in 4.321s
2022-11-23T01:51:49.1780862Z 
2022-11-23T01:51:49.1780967Z OK (skipped=1)
2022-11-23T01:51:49.1780987Z 
2022-11-23T01:51:49.1781113Z Generating XML reports...
2022-11-23T01:51:49.1781561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013148.xml
2022-11-23T01:51:49.1781932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1782108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1782484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1782672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1782697Z 
2022-11-23T01:51:49.1782787Z Running tests...
2022-11-23T01:51:49.1783051Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1783361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1783615Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-11-23T01:51:49.1783635Z 
2022-11-23T01:51:49.1783895Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1784005Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1784025Z 
2022-11-23T01:51:49.1784130Z OK (skipped=1)
2022-11-23T01:51:49.1784149Z 
2022-11-23T01:51:49.1784267Z Generating XML reports...
2022-11-23T01:51:49.1784715Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013155.xml
2022-11-23T01:51:49.1785064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1785245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1785621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1785862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1785883Z 
2022-11-23T01:51:49.1785992Z Running tests...
2022-11-23T01:51:49.1786254Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1786564Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1787049Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1787268Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10377
2022-11-23T01:51:49.1787484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10378
2022-11-23T01:51:49.1787915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1788096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1788479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1788670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1789069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1789245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1789620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1789795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1790149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1790393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1790796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1791196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1791424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1791652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1791753Z ok (4.963s)
2022-11-23T01:51:49.1791774Z 
2022-11-23T01:51:49.1792023Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1792134Z Ran 1 test in 4.964s
2022-11-23T01:51:49.1792158Z 
2022-11-23T01:51:49.1792250Z OK
2022-11-23T01:51:49.1792270Z 
2022-11-23T01:51:49.1792392Z Generating XML reports...
2022-11-23T01:51:49.1792845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013157.xml
2022-11-23T01:51:49.1793216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1793389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1793765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1793954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1793974Z 
2022-11-23T01:51:49.1794064Z Running tests...
2022-11-23T01:51:49.1794326Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1794637Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1794892Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1795109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10486
2022-11-23T01:51:49.1795324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10487
2022-11-23T01:51:49.1795777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1795965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1796331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1796520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1796886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1797062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1797438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1797623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1797870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1798112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1798510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1798888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1799115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1799407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1799508Z ok (6.558s)
2022-11-23T01:51:49.1799528Z 
2022-11-23T01:51:49.1799794Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1799902Z Ran 1 test in 6.558s
2022-11-23T01:51:49.1799922Z 
2022-11-23T01:51:49.1800017Z OK
2022-11-23T01:51:49.1800037Z 
2022-11-23T01:51:49.1800158Z Generating XML reports...
2022-11-23T01:51:49.1800593Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013205.xml
2022-11-23T01:51:49.1800957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1801130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1801504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1801698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1801718Z 
2022-11-23T01:51:49.1801825Z Running tests...
2022-11-23T01:51:49.1802086Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1802402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1802663Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1802863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10597
2022-11-23T01:51:49.1803079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10598
2022-11-23T01:51:49.1803448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1803622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1804002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1804189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1804554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1804781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1805173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1805345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1805590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1805832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1806234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1806637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1806865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1807095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1807335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1807556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1807952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1808341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1808500Z ok (5.007s)
2022-11-23T01:51:49.1808520Z 
2022-11-23T01:51:49.1808787Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1808898Z Ran 1 test in 5.007s
2022-11-23T01:51:49.1808919Z 
2022-11-23T01:51:49.1809010Z OK
2022-11-23T01:51:49.1809030Z 
2022-11-23T01:51:49.1809151Z Generating XML reports...
2022-11-23T01:51:49.1809608Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013214.xml
2022-11-23T01:51:49.1809964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1810140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1810517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1810708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1810731Z 
2022-11-23T01:51:49.1810838Z Running tests...
2022-11-23T01:51:49.1811102Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1811412Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1811681Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1811899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10712
2022-11-23T01:51:49.1812098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10713
2022-11-23T01:51:49.1812465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1812638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1813014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1813207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1813573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1813746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1814172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1814352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1814596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1814838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1815240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1815640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1815867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1816099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1816256Z skip: Skipped due to small world size. (4.228s)
2022-11-23T01:51:49.1816277Z 
2022-11-23T01:51:49.1816542Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1816636Z Ran 1 test in 4.228s
2022-11-23T01:51:49.1816656Z 
2022-11-23T01:51:49.1816763Z OK (skipped=1)
2022-11-23T01:51:49.1816782Z 
2022-11-23T01:51:49.1816902Z Generating XML reports...
2022-11-23T01:51:49.1817355Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013221.xml
2022-11-23T01:51:49.1817722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1817959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1818344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1818538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1818559Z 
2022-11-23T01:51:49.1818666Z Running tests...
2022-11-23T01:51:49.1818911Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1819219Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1819470Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1819687Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10821
2022-11-23T01:51:49.1819904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10822
2022-11-23T01:51:49.1820282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1820457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1820837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1821009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1821376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1821548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1821923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1822110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1822357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1822602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1823054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1823460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1823672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1823897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1824054Z skip: Skipped due to small world size. (4.153s)
2022-11-23T01:51:49.1824074Z 
2022-11-23T01:51:49.1824334Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1824445Z Ran 1 test in 4.153s
2022-11-23T01:51:49.1824465Z 
2022-11-23T01:51:49.1824569Z OK (skipped=1)
2022-11-23T01:51:49.1824589Z 
2022-11-23T01:51:49.1824710Z Generating XML reports...
2022-11-23T01:51:49.1825162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013228.xml
2022-11-23T01:51:49.1825518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1825695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1826073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1826261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1826282Z 
2022-11-23T01:51:49.1826390Z Running tests...
2022-11-23T01:51:49.1826653Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1827382Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1827647Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1827865Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10930
2022-11-23T01:51:49.1828070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10931
2022-11-23T01:51:49.1828442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1828617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1828991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1829180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1829542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1829721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1830099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1830276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1830524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1830768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1831174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1831568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1831804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1832029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1832188Z skip: Skipped due to small world size. (4.272s)
2022-11-23T01:51:49.1832211Z 
2022-11-23T01:51:49.1832561Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1832669Z Ran 1 test in 4.273s
2022-11-23T01:51:49.1832707Z 
2022-11-23T01:51:49.1832796Z OK (skipped=1)
2022-11-23T01:51:49.1832815Z 
2022-11-23T01:51:49.1832937Z Generating XML reports...
2022-11-23T01:51:49.1833392Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013235.xml
2022-11-23T01:51:49.1833759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1833935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1834319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1834509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1834530Z 
2022-11-23T01:51:49.1834637Z Running tests...
2022-11-23T01:51:49.1834882Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1835197Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1835466Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1835682Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11039
2022-11-23T01:51:49.1835898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11040
2022-11-23T01:51:49.1836265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1836518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1836902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1837074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1837443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1837617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1837989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1838175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1838417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1838660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1839063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1839459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1839674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1839903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1840137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1840376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1840773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1841168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1841268Z ok (5.220s)
2022-11-23T01:51:49.1841289Z 
2022-11-23T01:51:49.1841550Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1841663Z Ran 1 test in 5.220s
2022-11-23T01:51:49.1841748Z 
2022-11-23T01:51:49.1841829Z OK
2022-11-23T01:51:49.1841848Z 
2022-11-23T01:51:49.1841972Z Generating XML reports...
2022-11-23T01:51:49.1842432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013242.xml
2022-11-23T01:51:49.1842804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1842979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1843353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1843548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1843568Z 
2022-11-23T01:51:49.1843674Z Running tests...
2022-11-23T01:51:49.1843938Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1844236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1844646Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2022-11-23T01:51:49.1844666Z 
2022-11-23T01:51:49.1844926Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1845037Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1845057Z 
2022-11-23T01:51:49.1845163Z OK (skipped=1)
2022-11-23T01:51:49.1845182Z 
2022-11-23T01:51:49.1845303Z Generating XML reports...
2022-11-23T01:51:49.1845815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013249.xml
2022-11-23T01:51:49.1846183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1846362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1846724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1846915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1846934Z 
2022-11-23T01:51:49.1847040Z Running tests...
2022-11-23T01:51:49.1847299Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1847610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1847874Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1848099Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11187
2022-11-23T01:51:49.1848312Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11188
2022-11-23T01:51:49.1848668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1848845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1849226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1849415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1849782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1849956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1850332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1850519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1850762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1851044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1851455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1851848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1852075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1852303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1852465Z skip: Skipped due to small world size. (4.238s)
2022-11-23T01:51:49.1852486Z 
2022-11-23T01:51:49.1852752Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1852863Z Ran 1 test in 4.238s
2022-11-23T01:51:49.1852882Z 
2022-11-23T01:51:49.1852987Z OK (skipped=1)
2022-11-23T01:51:49.1853007Z 
2022-11-23T01:51:49.1853117Z Generating XML reports...
2022-11-23T01:51:49.1853569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013252.xml
2022-11-23T01:51:49.1853938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1854111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1854487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1854676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1854749Z 
2022-11-23T01:51:49.1854862Z Running tests...
2022-11-23T01:51:49.1855128Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1855423Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1855693Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1855911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11296
2022-11-23T01:51:49.1856126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11297
2022-11-23T01:51:49.1856494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1856669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1857044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1857238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1857604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1857764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1858143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1858329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1858569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1858810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1859207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1859608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1859837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1860113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1860222Z ok (4.268s)
2022-11-23T01:51:49.1860242Z 
2022-11-23T01:51:49.1860510Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1860622Z Ran 1 test in 4.269s
2022-11-23T01:51:49.1860642Z 
2022-11-23T01:51:49.1860731Z OK
2022-11-23T01:51:49.1860750Z 
2022-11-23T01:51:49.1860870Z Generating XML reports...
2022-11-23T01:51:49.1861319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013258.xml
2022-11-23T01:51:49.1861688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1861868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1862231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1862423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1862443Z 
2022-11-23T01:51:49.1862548Z Running tests...
2022-11-23T01:51:49.1862813Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1863125Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1863395Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1863612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11405
2022-11-23T01:51:49.1863828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11406
2022-11-23T01:51:49.1864247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1864423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1864801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1864991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1865356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1865528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1865901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1866087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1866335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1866563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1867285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1867713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1867943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1868171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1868273Z ok (10.351s)
2022-11-23T01:51:49.1868294Z 
2022-11-23T01:51:49.1868555Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1868673Z Ran 1 test in 10.351s
2022-11-23T01:51:49.1868693Z 
2022-11-23T01:51:49.1868784Z OK
2022-11-23T01:51:49.1868804Z 
2022-11-23T01:51:49.1868910Z Generating XML reports...
2022-11-23T01:51:49.1869363Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013305.xml
2022-11-23T01:51:49.1869834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1870021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1870400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1870588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1870608Z 
2022-11-23T01:51:49.1870715Z Running tests...
2022-11-23T01:51:49.1870978Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1871272Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1871551Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:51:49.1871570Z 
2022-11-23T01:51:49.1871827Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1871941Z Ran 1 test in 0.003s
2022-11-23T01:51:49.1871961Z 
2022-11-23T01:51:49.1872066Z OK (skipped=1)
2022-11-23T01:51:49.1872085Z 
2022-11-23T01:51:49.1872207Z Generating XML reports...
2022-11-23T01:51:49.1872650Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013323.xml
2022-11-23T01:51:49.1873014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1873188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1873544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1873813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1873832Z 
2022-11-23T01:51:49.1873938Z Running tests...
2022-11-23T01:51:49.1874200Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1874510Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1874768Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-11-23T01:51:49.1874788Z 
2022-11-23T01:51:49.1875047Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1875157Z Ran 1 test in 0.003s
2022-11-23T01:51:49.1875177Z 
2022-11-23T01:51:49.1875283Z OK (skipped=1)
2022-11-23T01:51:49.1875302Z 
2022-11-23T01:51:49.1875407Z Generating XML reports...
2022-11-23T01:51:49.1875852Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013325.xml
2022-11-23T01:51:49.1876226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1876400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1876784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1876976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1876996Z 
2022-11-23T01:51:49.1877105Z Running tests...
2022-11-23T01:51:49.1877365Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1877657Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1877926Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.005s)
2022-11-23T01:51:49.1877950Z 
2022-11-23T01:51:49.1878204Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1878314Z Ran 1 test in 0.006s
2022-11-23T01:51:49.1878333Z 
2022-11-23T01:51:49.1878439Z OK (skipped=1)
2022-11-23T01:51:49.1878458Z 
2022-11-23T01:51:49.1878580Z Generating XML reports...
2022-11-23T01:51:49.1879073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013327.xml
2022-11-23T01:51:49.1879452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1879628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1879985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1880177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1880201Z 
2022-11-23T01:51:49.1880308Z Running tests...
2022-11-23T01:51:49.1880571Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1880880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1881140Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:51:49.1881161Z 
2022-11-23T01:51:49.1881419Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1881531Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1881550Z 
2022-11-23T01:51:49.1881660Z OK (skipped=1)
2022-11-23T01:51:49.1881679Z 
2022-11-23T01:51:49.1881784Z Generating XML reports...
2022-11-23T01:51:49.1882232Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013330.xml
2022-11-23T01:51:49.1882599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1882842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1883224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1883414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1883434Z 
2022-11-23T01:51:49.1883546Z Running tests...
2022-11-23T01:51:49.1883807Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1884102Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1884366Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:51:49.1884386Z 
2022-11-23T01:51:49.1884643Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1884752Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1884776Z 
2022-11-23T01:51:49.1884884Z OK (skipped=1)
2022-11-23T01:51:49.1884903Z 
2022-11-23T01:51:49.1885026Z Generating XML reports...
2022-11-23T01:51:49.1885472Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013332.xml
2022-11-23T01:51:49.1885845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1886025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1886385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1886575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1886595Z 
2022-11-23T01:51:49.1886702Z Running tests...
2022-11-23T01:51:49.1886960Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1887270Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1887550Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:51:49.1887570Z 
2022-11-23T01:51:49.1887826Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1887938Z Ran 1 test in 0.003s
2022-11-23T01:51:49.1888009Z 
2022-11-23T01:51:49.1888124Z OK (skipped=1)
2022-11-23T01:51:49.1888144Z 
2022-11-23T01:51:49.1888248Z Generating XML reports...
2022-11-23T01:51:49.1888694Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013335.xml
2022-11-23T01:51:49.1889111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1889286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1889659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1889851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1889870Z 
2022-11-23T01:51:49.1889979Z Running tests...
2022-11-23T01:51:49.1890237Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1890549Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1890798Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:51:49.1890818Z 
2022-11-23T01:51:49.1891077Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1891186Z Ran 1 test in 0.003s
2022-11-23T01:51:49.1891205Z 
2022-11-23T01:51:49.1891311Z OK (skipped=1)
2022-11-23T01:51:49.1891330Z 
2022-11-23T01:51:49.1891451Z Generating XML reports...
2022-11-23T01:51:49.1891899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013337.xml
2022-11-23T01:51:49.1892394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1892570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1892950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1893123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1893142Z 
2022-11-23T01:51:49.1893249Z Running tests...
2022-11-23T01:51:49.1893507Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1893814Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1894076Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T01:51:49.1894100Z 
2022-11-23T01:51:49.1894357Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1894467Z Ran 1 test in 0.002s
2022-11-23T01:51:49.1894486Z 
2022-11-23T01:51:49.1894592Z OK (skipped=1)
2022-11-23T01:51:49.1894611Z 
2022-11-23T01:51:49.1894716Z Generating XML reports...
2022-11-23T01:51:49.1895162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013339.xml
2022-11-23T01:51:49.1895528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1895704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1896077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1896267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1896287Z 
2022-11-23T01:51:49.1896398Z Running tests...
2022-11-23T01:51:49.1896660Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1896972Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1897200Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1897474Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11778
2022-11-23T01:51:49.1897698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11779
2022-11-23T01:51:49.1898071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1898244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1898617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1898807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1899175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1899332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1899710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1899899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1900143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1900387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1900791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1901186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1901474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1901816Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1902032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1902364Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1902699Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1903253Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1903274Z 
2022-11-23T01:51:49.1903620Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1903948Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1904266Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1904600Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1904925Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1905268Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1905592Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1905913Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1906233Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1906559Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1907132Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1907576Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1907938Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1908256Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1908574Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1908880Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1909224Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1909556Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1909893Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1910213Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1910531Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1910855Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1911408Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1911495Z 
2022-11-23T01:51:49.1911857Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1912186Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1912495Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1912827Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1913153Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1913496Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1913835Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1914155Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1914480Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1914803Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1915146Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1915463Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1915800Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1916118Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1916434Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1916758Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1917316Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1917337Z 
2022-11-23T01:51:49.1917734Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1918073Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1918393Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1918708Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1919051Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1919386Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1919726Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1920047Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1920367Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1920694Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1921016Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1921357Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1921676Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1922067Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1922379Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1922709Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1923031Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1923368Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1923701Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1924018Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1924333Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1924644Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1924966Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1925305Z STAGE:2022-11-23 01:33:46 11779:11779 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1925642Z STAGE:2022-11-23 01:33:46 11778:11778 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1925744Z ok (4.363s)
2022-11-23T01:51:49.1925764Z 
2022-11-23T01:51:49.1926026Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1926137Z Ran 1 test in 4.364s
2022-11-23T01:51:49.1926157Z 
2022-11-23T01:51:49.1926250Z OK
2022-11-23T01:51:49.1926270Z 
2022-11-23T01:51:49.1926392Z Generating XML reports...
2022-11-23T01:51:49.1926827Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013342.xml
2022-11-23T01:51:49.1927203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1927378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1927820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1928020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1928040Z 
2022-11-23T01:51:49.1928147Z Running tests...
2022-11-23T01:51:49.1928416Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1928728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1928967Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1929717Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2022-11-23T01:51:49.1929761Z 
2022-11-23T01:51:49.1930010Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1930121Z Ran 1 test in 1.633s
2022-11-23T01:51:49.1930141Z 
2022-11-23T01:51:49.1930249Z OK (skipped=1)
2022-11-23T01:51:49.1930268Z 
2022-11-23T01:51:49.1930389Z Generating XML reports...
2022-11-23T01:51:49.1930839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013349.xml
2022-11-23T01:51:49.1931209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1931385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1931834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1932006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1932044Z 
2022-11-23T01:51:49.1932134Z Running tests...
2022-11-23T01:51:49.1932394Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1932704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1932967Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1933186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11925
2022-11-23T01:51:49.1933401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11926
2022-11-23T01:51:49.1933771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1933934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1934311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1934502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1934868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1935043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1935419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1935608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1935854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1936102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1936484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1936883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1937163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1937398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1937639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.1937876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.1938273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1938604Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1938984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.1939301Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1939841Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1939862Z 
2022-11-23T01:51:49.1940426Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1940446Z 
2022-11-23T01:51:49.1940769Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1941155Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1941487Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1942036Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1942056Z 
2022-11-23T01:51:49.1942403Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1942728Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1943046Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1943381Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1943933Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1943954Z 
2022-11-23T01:51:49.1944286Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1944612Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1944932Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1945261Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1945586Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1945935Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1946273Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1946591Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1947172Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1947521Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1947847Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1948192Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1948529Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1948852Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1949166Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1949494Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1949817Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1950155Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1950473Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1950791Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1951106Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1951515Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1951838Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1952178Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1952517Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1952836Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1953152Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1953458Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1953785Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1954127Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1954463Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1954784Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1955102Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1955425Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1955745Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1956081Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1956406Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1956724Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1957040Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1957417Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1957981Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1958002Z 
2022-11-23T01:51:49.1958344Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1958667Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1958990Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1959321Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1959878Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1959898Z 
2022-11-23T01:51:49.1960243Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1960552Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1960872Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.1961201Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1961626Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.1961970Z STAGE:2022-11-23 01:33:57 11926:11926 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1962310Z STAGE:2022-11-23 01:33:57 11925:11925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.1962413Z ok (4.325s)
2022-11-23T01:51:49.1962433Z 
2022-11-23T01:51:49.1962700Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1962811Z Ran 1 test in 4.326s
2022-11-23T01:51:49.1962831Z 
2022-11-23T01:51:49.1962905Z OK
2022-11-23T01:51:49.1962924Z 
2022-11-23T01:51:49.1963047Z Generating XML reports...
2022-11-23T01:51:49.1963500Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013353.xml
2022-11-23T01:51:49.1963871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1964053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1964434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1964630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1964650Z 
2022-11-23T01:51:49.1964758Z Running tests...
2022-11-23T01:51:49.1965004Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1965313Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1965569Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1965788Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12044
2022-11-23T01:51:49.1966004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12045
2022-11-23T01:51:49.1966377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1966555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1966983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1967184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1967539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1967715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1968087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1968279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1968528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1968771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1969172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1969573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1969785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1970013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1970170Z skip: Skipped due to small world size. (4.275s)
2022-11-23T01:51:49.1970191Z 
2022-11-23T01:51:49.1970456Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1970626Z Ran 1 test in 4.275s
2022-11-23T01:51:49.1970646Z 
2022-11-23T01:51:49.1970754Z OK (skipped=1)
2022-11-23T01:51:49.1970773Z 
2022-11-23T01:51:49.1970896Z Generating XML reports...
2022-11-23T01:51:49.1971350Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013400.xml
2022-11-23T01:51:49.1971724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1971883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1972262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1972452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1972472Z 
2022-11-23T01:51:49.1972579Z Running tests...
2022-11-23T01:51:49.1972838Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1973154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1973419Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1973635Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12153
2022-11-23T01:51:49.1973839Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12154
2022-11-23T01:51:49.1974211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1974385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1974756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1974944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1975308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1975486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1975861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1976105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1976342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1976585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1976990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1977384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1977613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1977845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1978618Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:51:49.1978733Z   warnings.warn(
2022-11-23T01:51:49.1979489Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T01:51:49.1979600Z   warnings.warn(
2022-11-23T01:51:49.1979683Z ok (5.751s)
2022-11-23T01:51:49.1979703Z 
2022-11-23T01:51:49.1980031Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1980146Z Ran 1 test in 5.751s
2022-11-23T01:51:49.1980166Z 
2022-11-23T01:51:49.1980258Z OK
2022-11-23T01:51:49.1980277Z 
2022-11-23T01:51:49.1980403Z Generating XML reports...
2022-11-23T01:51:49.1980856Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013406.xml
2022-11-23T01:51:49.1981224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1981400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1981761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1981954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1981974Z 
2022-11-23T01:51:49.1982080Z Running tests...
2022-11-23T01:51:49.1982348Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1982659Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1982922Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1983668Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-11-23T01:51:49.1983689Z 
2022-11-23T01:51:49.1983950Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1984061Z Ran 1 test in 1.642s
2022-11-23T01:51:49.1984080Z 
2022-11-23T01:51:49.1984185Z OK (skipped=1)
2022-11-23T01:51:49.1984204Z 
2022-11-23T01:51:49.1984309Z Generating XML reports...
2022-11-23T01:51:49.1984764Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013415.xml
2022-11-23T01:51:49.1985136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1985309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1985737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1985937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1985957Z 
2022-11-23T01:51:49.1986065Z Running tests...
2022-11-23T01:51:49.1986329Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1986624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1987126Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1987905Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.655s)
2022-11-23T01:51:49.1987927Z 
2022-11-23T01:51:49.1988185Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1988296Z Ran 1 test in 1.655s
2022-11-23T01:51:49.1988316Z 
2022-11-23T01:51:49.1988422Z OK (skipped=1)
2022-11-23T01:51:49.1988441Z 
2022-11-23T01:51:49.1988563Z Generating XML reports...
2022-11-23T01:51:49.1989039Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013419.xml
2022-11-23T01:51:49.1989413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1989684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1990049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1990240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1990264Z 
2022-11-23T01:51:49.1990371Z Running tests...
2022-11-23T01:51:49.1990631Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1990943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1991260Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1991999Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2022-11-23T01:51:49.1992024Z 
2022-11-23T01:51:49.1992285Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1992396Z Ran 1 test in 1.621s
2022-11-23T01:51:49.1992416Z 
2022-11-23T01:51:49.1992526Z OK (skipped=1)
2022-11-23T01:51:49.1992545Z 
2022-11-23T01:51:49.1992651Z Generating XML reports...
2022-11-23T01:51:49.1993097Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013423.xml
2022-11-23T01:51:49.1993465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1993638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1994017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1994214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1994234Z 
2022-11-23T01:51:49.1994340Z Running tests...
2022-11-23T01:51:49.1994600Z ----------------------------------------------------------------------
2022-11-23T01:51:49.1994992Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.1995253Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.1995474Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12366
2022-11-23T01:51:49.1995690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12367
2022-11-23T01:51:49.1996069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1996244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1996627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1996817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1997182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.1997342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.1997720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.1997909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.1998156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.1998400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.1998799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1999262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.1999492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.1999723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.1999963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7yxz9vr
2022-11-23T01:51:49.2000234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7yxz9vr/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2000486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2rk5eyh
2022-11-23T01:51:49.2000755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2rk5eyh/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2000995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2001231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2001332Z ok (6.256s)
2022-11-23T01:51:49.2001352Z 
2022-11-23T01:51:49.2001622Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2001720Z Ran 1 test in 6.256s
2022-11-23T01:51:49.2001757Z 
2022-11-23T01:51:49.2001832Z OK
2022-11-23T01:51:49.2001851Z 
2022-11-23T01:51:49.2001973Z Generating XML reports...
2022-11-23T01:51:49.2002423Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013427.xml
2022-11-23T01:51:49.2002793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2002970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2003349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2003543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2003563Z 
2022-11-23T01:51:49.2003672Z Running tests...
2022-11-23T01:51:49.2003916Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2004277Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2004562Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2004779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12481
2022-11-23T01:51:49.2004998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12482
2022-11-23T01:51:49.2005372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2005551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2005929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2006102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2006470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2006646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2007021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2007210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2007456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2007700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2008167Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2008562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2008779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2009007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2009342Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwuzidq3z
2022-11-23T01:51:49.2009614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwuzidq3z/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2009867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy3gjdq29
2022-11-23T01:51:49.2010132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy3gjdq29/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2010370Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2010605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2010841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2011057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2011157Z ok (6.262s)
2022-11-23T01:51:49.2011178Z 
2022-11-23T01:51:49.2011447Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2011559Z Ran 1 test in 6.263s
2022-11-23T01:51:49.2011579Z 
2022-11-23T01:51:49.2011672Z OK
2022-11-23T01:51:49.2011692Z 
2022-11-23T01:51:49.2011813Z Generating XML reports...
2022-11-23T01:51:49.2012262Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013436.xml
2022-11-23T01:51:49.2012637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2012795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2013226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2013428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2013448Z 
2022-11-23T01:51:49.2013556Z Running tests...
2022-11-23T01:51:49.2013821Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2014130Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2014400Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2015138Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.605s)
2022-11-23T01:51:49.2015164Z 
2022-11-23T01:51:49.2015426Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2015539Z Ran 1 test in 1.606s
2022-11-23T01:51:49.2015559Z 
2022-11-23T01:51:49.2015648Z OK (skipped=1)
2022-11-23T01:51:49.2015667Z 
2022-11-23T01:51:49.2015790Z Generating XML reports...
2022-11-23T01:51:49.2016233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013445.xml
2022-11-23T01:51:49.2016599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2016776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2017219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2017410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2017431Z 
2022-11-23T01:51:49.2017539Z Running tests...
2022-11-23T01:51:49.2017784Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2018093Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2018380Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2019118Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.641s)
2022-11-23T01:51:49.2019142Z 
2022-11-23T01:51:49.2019400Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2019509Z Ran 1 test in 1.641s
2022-11-23T01:51:49.2019528Z 
2022-11-23T01:51:49.2019635Z OK (skipped=1)
2022-11-23T01:51:49.2019654Z 
2022-11-23T01:51:49.2019776Z Generating XML reports...
2022-11-23T01:51:49.2020223Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013449.xml
2022-11-23T01:51:49.2020590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2020747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2021120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2021308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2021327Z 
2022-11-23T01:51:49.2021439Z Running tests...
2022-11-23T01:51:49.2021697Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2022003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2022288Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2022567Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12664
2022-11-23T01:51:49.2022796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12665
2022-11-23T01:51:49.2023154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2023331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2023708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2023902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2024264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2024441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2024822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2025012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2025240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2025484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2025884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2026278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2026568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2026998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2027277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpka0x62y_
2022-11-23T01:51:49.2027546Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpka0x62y_/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2027802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qkdpe8_
2022-11-23T01:51:49.2028054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qkdpe8_/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2028275Z <class 'torch.nn.parameter.Parameter'>
2022-11-23T01:51:49.2028376Z ok (5.746s)
2022-11-23T01:51:49.2028396Z 
2022-11-23T01:51:49.2028665Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2028776Z Ran 1 test in 5.746s
2022-11-23T01:51:49.2028796Z 
2022-11-23T01:51:49.2028888Z OK
2022-11-23T01:51:49.2028907Z 
2022-11-23T01:51:49.2029030Z Generating XML reports...
2022-11-23T01:51:49.2029482Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013453.xml
2022-11-23T01:51:49.2029837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2030017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2030396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2030584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2030604Z 
2022-11-23T01:51:49.2030710Z Running tests...
2022-11-23T01:51:49.2030980Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2031294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2031601Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2031905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12775
2022-11-23T01:51:49.2032118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12776
2022-11-23T01:51:49.2032496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2032669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2033053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2033246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2033618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2033793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2034173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2034343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2034587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2034831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2035230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2035625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2035928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2036156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2036415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7_c52dm
2022-11-23T01:51:49.2036686Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7_c52dm/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2036922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpro4zybgm
2022-11-23T01:51:49.2037190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpro4zybgm/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2037290Z ok (5.758s)
2022-11-23T01:51:49.2037311Z 
2022-11-23T01:51:49.2037582Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2037692Z Ran 1 test in 5.759s
2022-11-23T01:51:49.2037716Z 
2022-11-23T01:51:49.2037808Z OK
2022-11-23T01:51:49.2037827Z 
2022-11-23T01:51:49.2037949Z Generating XML reports...
2022-11-23T01:51:49.2038397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013501.xml
2022-11-23T01:51:49.2038772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2038931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2039311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2039499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2039519Z 
2022-11-23T01:51:49.2039627Z Running tests...
2022-11-23T01:51:49.2039888Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2040196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2040464Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2040682Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12886
2022-11-23T01:51:49.2040984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12887
2022-11-23T01:51:49.2041370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2041544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2041921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2042110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2042475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2042655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2043028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2043216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2043448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2043692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2044088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2044480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2044709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2044996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2045251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8u1i7pm6
2022-11-23T01:51:49.2045522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8u1i7pm6/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2045773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo504n1dn
2022-11-23T01:51:49.2046022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo504n1dn/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2046254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2046488Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2046589Z ok (6.210s)
2022-11-23T01:51:49.2046610Z 
2022-11-23T01:51:49.2046881Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2046997Z Ran 1 test in 6.210s
2022-11-23T01:51:49.2047017Z 
2022-11-23T01:51:49.2047109Z OK
2022-11-23T01:51:49.2047129Z 
2022-11-23T01:51:49.2047250Z Generating XML reports...
2022-11-23T01:51:49.2047683Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013510.xml
2022-11-23T01:51:49.2048057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2048234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2048614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2048805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2048825Z 
2022-11-23T01:51:49.2048932Z Running tests...
2022-11-23T01:51:49.2049191Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2049507Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2049779Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2050050Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13001
2022-11-23T01:51:49.2050279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13002
2022-11-23T01:51:49.2050650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2050825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2051205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2051396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2051767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2051942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2052297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2052489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2052732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2052975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2053373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2053766Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2054054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2054282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2054537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqht9z7m
2022-11-23T01:51:49.2054793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqht9z7m/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2055045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8rqo82_
2022-11-23T01:51:49.2055308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8rqo82_/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2056089Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:51:49.2056200Z ok (6.356s)
2022-11-23T01:51:49.2056220Z 
2022-11-23T01:51:49.2056496Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2056607Z Ran 1 test in 6.357s
2022-11-23T01:51:49.2056627Z 
2022-11-23T01:51:49.2056719Z OK
2022-11-23T01:51:49.2056739Z 
2022-11-23T01:51:49.2056861Z Generating XML reports...
2022-11-23T01:51:49.2057312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013518.xml
2022-11-23T01:51:49.2057664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2057841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2058227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2058417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2058437Z 
2022-11-23T01:51:49.2058543Z Running tests...
2022-11-23T01:51:49.2058873Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2059200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2059482Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2060228Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.650s)
2022-11-23T01:51:49.2060254Z 
2022-11-23T01:51:49.2060515Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2060609Z Ran 1 test in 1.650s
2022-11-23T01:51:49.2060628Z 
2022-11-23T01:51:49.2060734Z OK (skipped=1)
2022-11-23T01:51:49.2060753Z 
2022-11-23T01:51:49.2060875Z Generating XML reports...
2022-11-23T01:51:49.2061322Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013527.xml
2022-11-23T01:51:49.2061693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2061868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2062243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2062433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2062511Z 
2022-11-23T01:51:49.2062629Z Running tests...
2022-11-23T01:51:49.2062876Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2063187Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2063449Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2063669Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13150
2022-11-23T01:51:49.2063885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13151
2022-11-23T01:51:49.2064256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2064431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2064805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2064983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2065351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2065525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2065901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2066090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2066334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2066576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2067159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2067579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2067792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2068019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2068359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7fxdp9s
2022-11-23T01:51:49.2068648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7fxdp9s/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2068904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpncxfn0ti
2022-11-23T01:51:49.2069172Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpncxfn0ti/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2070079Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:51:49.2070968Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:51:49.2072135Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T01:51:49.2072447Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-11-23T01:51:49.2073608Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T01:51:49.2073839Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-11-23T01:51:49.2074080Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2074316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2075205Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:51:49.2076082Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:51:49.2076955Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:51:49.2077868Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:51:49.2078742Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:51:49.2079610Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:51:49.2080465Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:51:49.2081320Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:51:49.2082242Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:51:49.2083095Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T01:51:49.2083202Z ok (4.221s)
2022-11-23T01:51:49.2083223Z 
2022-11-23T01:51:49.2083488Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2083583Z Ran 1 test in 4.221s
2022-11-23T01:51:49.2083603Z 
2022-11-23T01:51:49.2083697Z OK
2022-11-23T01:51:49.2083717Z 
2022-11-23T01:51:49.2083842Z Generating XML reports...
2022-11-23T01:51:49.2084293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013531.xml
2022-11-23T01:51:49.2084663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2084837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2085216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2085408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2085433Z 
2022-11-23T01:51:49.2085523Z Running tests...
2022-11-23T01:51:49.2085790Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2086099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2086397Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2087149Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.615s)
2022-11-23T01:51:49.2087170Z 
2022-11-23T01:51:49.2087426Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2087537Z Ran 1 test in 1.616s
2022-11-23T01:51:49.2087561Z 
2022-11-23T01:51:49.2087668Z OK (skipped=1)
2022-11-23T01:51:49.2087687Z 
2022-11-23T01:51:49.2087813Z Generating XML reports...
2022-11-23T01:51:49.2088319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013538.xml
2022-11-23T01:51:49.2088675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2088879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2089264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2089457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2089476Z 
2022-11-23T01:51:49.2089584Z Running tests...
2022-11-23T01:51:49.2089842Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2090153Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2090486Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2090686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13297
2022-11-23T01:51:49.2090905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13298
2022-11-23T01:51:49.2091281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2091456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2091837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2092027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2092389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2092568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2092945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2093115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2093367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2093612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2094013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2094409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2094638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2094869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2095126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63hkmv51
2022-11-23T01:51:49.2095397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63hkmv51/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2095691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0bo5_4kt
2022-11-23T01:51:49.2095970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0bo5_4kt/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2096764Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T01:51:49.2097096Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T01:51:49.2097881Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T01:51:49.2098212Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T01:51:49.2098314Z ok (6.258s)
2022-11-23T01:51:49.2098334Z 
2022-11-23T01:51:49.2098596Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2098708Z Ran 1 test in 6.259s
2022-11-23T01:51:49.2098728Z 
2022-11-23T01:51:49.2098820Z OK
2022-11-23T01:51:49.2098839Z 
2022-11-23T01:51:49.2098944Z Generating XML reports...
2022-11-23T01:51:49.2099393Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013542.xml
2022-11-23T01:51:49.2099830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2100007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2100388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2100578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2100598Z 
2022-11-23T01:51:49.2100705Z Running tests...
2022-11-23T01:51:49.2100968Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2101282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2101539Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2102287Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.607s)
2022-11-23T01:51:49.2102307Z 
2022-11-23T01:51:49.2102570Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2102681Z Ran 1 test in 1.608s
2022-11-23T01:51:49.2102701Z 
2022-11-23T01:51:49.2102807Z OK (skipped=1)
2022-11-23T01:51:49.2102826Z 
2022-11-23T01:51:49.2102950Z Generating XML reports...
2022-11-23T01:51:49.2103397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013551.xml
2022-11-23T01:51:49.2103764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2103938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2104304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2104495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2104514Z 
2022-11-23T01:51:49.2104621Z Running tests...
2022-11-23T01:51:49.2104931Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2105258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2105528Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2106271Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2022-11-23T01:51:49.2106297Z 
2022-11-23T01:51:49.2106557Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2106668Z Ran 1 test in 1.634s
2022-11-23T01:51:49.2106687Z 
2022-11-23T01:51:49.2106974Z OK (skipped=1)
2022-11-23T01:51:49.2106997Z 
2022-11-23T01:51:49.2107114Z Generating XML reports...
2022-11-23T01:51:49.2107574Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013555.xml
2022-11-23T01:51:49.2107942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2108116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2108492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2108682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2108785Z 
2022-11-23T01:51:49.2108901Z Running tests...
2022-11-23T01:51:49.2109167Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2109477Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2109752Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2109970Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13510
2022-11-23T01:51:49.2110187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13511
2022-11-23T01:51:49.2110556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2110730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2111108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2111303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2111672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2111829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2112210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2112400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2112644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2112890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2113290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2113690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2113919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2114147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2114438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2114689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2115088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2115474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2115730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoeyruhfz
2022-11-23T01:51:49.2116010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoeyruhfz/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2116262Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpya1u6abp
2022-11-23T01:51:49.2116531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpya1u6abp/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2116765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2116982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2117215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2117447Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2117547Z ok (6.618s)
2022-11-23T01:51:49.2117567Z 
2022-11-23T01:51:49.2117833Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2118003Z Ran 1 test in 6.618s
2022-11-23T01:51:49.2118022Z 
2022-11-23T01:51:49.2118114Z OK
2022-11-23T01:51:49.2118133Z 
2022-11-23T01:51:49.2118256Z Generating XML reports...
2022-11-23T01:51:49.2118699Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013559.xml
2022-11-23T01:51:49.2119075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2119250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2119629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2119819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2119839Z 
2022-11-23T01:51:49.2119945Z Running tests...
2022-11-23T01:51:49.2120205Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2120525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2120802Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2121006Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13631
2022-11-23T01:51:49.2121223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13632
2022-11-23T01:51:49.2121595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2121768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2122145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2122335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2122709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2122883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2123240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2123499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2123756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2124161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2124404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2124798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2125031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2125308Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:51:49.2125534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2125786Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:51:49.2126044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2rp4j30
2022-11-23T01:51:49.2126310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2rp4j30/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2126558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78tf75ya
2022-11-23T01:51:49.2126822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78tf75ya/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2127118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2127352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2127583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2127820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2128080Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:51:49.2128357Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:51:49.2128630Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:51:49.2128901Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T01:51:49.2129134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2129366Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2129596Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2129830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2130105Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:51:49.2130361Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T01:51:49.2130638Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T01:51:49.2130904Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T01:51:49.2131138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2131369Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2131646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2131885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2131986Z ok (7.145s)
2022-11-23T01:51:49.2132006Z 
2022-11-23T01:51:49.2132265Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2132377Z Ran 1 test in 7.145s
2022-11-23T01:51:49.2132396Z 
2022-11-23T01:51:49.2132488Z OK
2022-11-23T01:51:49.2132508Z 
2022-11-23T01:51:49.2132630Z Generating XML reports...
2022-11-23T01:51:49.2133080Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013609.xml
2022-11-23T01:51:49.2133458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2133634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2134019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2134210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2134230Z 
2022-11-23T01:51:49.2134320Z Running tests...
2022-11-23T01:51:49.2134581Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2134890Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2135160Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2135906Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.644s)
2022-11-23T01:51:49.2135982Z 
2022-11-23T01:51:49.2136249Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2136365Z Ran 1 test in 1.644s
2022-11-23T01:51:49.2136385Z 
2022-11-23T01:51:49.2136492Z OK (skipped=1)
2022-11-23T01:51:49.2136511Z 
2022-11-23T01:51:49.2136633Z Generating XML reports...
2022-11-23T01:51:49.2137063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013618.xml
2022-11-23T01:51:49.2137434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2137611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2137992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2138179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2138198Z 
2022-11-23T01:51:49.2138306Z Running tests...
2022-11-23T01:51:49.2138568Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2138880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2139155Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2139358Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13780
2022-11-23T01:51:49.2139573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13781
2022-11-23T01:51:49.2139943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2140123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2140501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2140690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2141107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2141288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2141667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2141838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2142084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2142330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2142734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2143128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2143361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2143909Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:51:49.2144137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2144736Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T01:51:49.2144997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2gkyj94n
2022-11-23T01:51:49.2145270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2gkyj94n/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2145508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyduepdu
2022-11-23T01:51:49.2145779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyduepdu/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2146013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2146247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2146526Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T01:51:49.2146984Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T01:51:49.2147305Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T01:51:49.2147605Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T01:51:49.2147936Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T01:51:49.2148248Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T01:51:49.2148574Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T01:51:49.2148890Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T01:51:49.2149205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2149453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2149554Z ok (6.352s)
2022-11-23T01:51:49.2149576Z 
2022-11-23T01:51:49.2149860Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2149970Z Ran 1 test in 6.352s
2022-11-23T01:51:49.2149990Z 
2022-11-23T01:51:49.2150083Z OK
2022-11-23T01:51:49.2150103Z 
2022-11-23T01:51:49.2150208Z Generating XML reports...
2022-11-23T01:51:49.2150660Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013622.xml
2022-11-23T01:51:49.2151037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2151221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2151606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2151799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2151819Z 
2022-11-23T01:51:49.2151927Z Running tests...
2022-11-23T01:51:49.2152192Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2152485Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2152799Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2153097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13895
2022-11-23T01:51:49.2153314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13896
2022-11-23T01:51:49.2153689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2153869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2154246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2154437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2154803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2154961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2155338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2155531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2155775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2156023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2156425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2156823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2157053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2157279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2157518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1akj9nhb
2022-11-23T01:51:49.2157792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1akj9nhb/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2158046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8mcn0pq
2022-11-23T01:51:49.2158365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8mcn0pq/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2158608Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2158843Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2159073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2159306Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2159390Z ok (6.455s)
2022-11-23T01:51:49.2159428Z 
2022-11-23T01:51:49.2159695Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2159807Z Ran 1 test in 6.455s
2022-11-23T01:51:49.2159828Z 
2022-11-23T01:51:49.2159920Z OK
2022-11-23T01:51:49.2159940Z 
2022-11-23T01:51:49.2160063Z Generating XML reports...
2022-11-23T01:51:49.2160521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013631.xml
2022-11-23T01:51:49.2160893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2161068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2161446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2161618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2161637Z 
2022-11-23T01:51:49.2161743Z Running tests...
2022-11-23T01:51:49.2162074Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2162383Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2162696Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2162921Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14040
2022-11-23T01:51:49.2163138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14041
2022-11-23T01:51:49.2163512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2163668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2164045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2164233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2164603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2164777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2165158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2165350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2165596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2165839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2166224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2166621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2166856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2167082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2167394Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphouel89b
2022-11-23T01:51:49.2167673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphouel89b/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2167929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppp0v5zw3
2022-11-23T01:51:49.2168196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppp0v5zw3/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2168431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2168647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2168885Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2169117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2169219Z ok (6.343s)
2022-11-23T01:51:49.2169240Z 
2022-11-23T01:51:49.2169516Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2169629Z Ran 1 test in 6.344s
2022-11-23T01:51:49.2169649Z 
2022-11-23T01:51:49.2169742Z OK
2022-11-23T01:51:49.2169761Z 
2022-11-23T01:51:49.2169883Z Generating XML reports...
2022-11-23T01:51:49.2170319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013640.xml
2022-11-23T01:51:49.2170689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2170867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2171316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2171510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2171529Z 
2022-11-23T01:51:49.2171637Z Running tests...
2022-11-23T01:51:49.2171902Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2172212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2172586Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2172788Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14185
2022-11-23T01:51:49.2173006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14186
2022-11-23T01:51:49.2173375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2173556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2173936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2174130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2174497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2174673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2175035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2175225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2175468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2175718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2176117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2176567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2176804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2177032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2177288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6dg0u0mk
2022-11-23T01:51:49.2177539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6dg0u0mk/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2177793Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9sreuk8_
2022-11-23T01:51:49.2178062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9sreuk8_/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2178295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2178531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2178763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2178995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2179096Z ok (6.441s)
2022-11-23T01:51:49.2179116Z 
2022-11-23T01:51:49.2179386Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2179482Z Ran 1 test in 6.441s
2022-11-23T01:51:49.2179501Z 
2022-11-23T01:51:49.2179593Z OK
2022-11-23T01:51:49.2179613Z 
2022-11-23T01:51:49.2179735Z Generating XML reports...
2022-11-23T01:51:49.2180302Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013649.xml
2022-11-23T01:51:49.2180674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2180848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2181229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2181421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2181441Z 
2022-11-23T01:51:49.2181530Z Running tests...
2022-11-23T01:51:49.2181796Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2182106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2182475Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2182697Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14330
2022-11-23T01:51:49.2182913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14331
2022-11-23T01:51:49.2183291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2183468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2183847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2184022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2184391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2184568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2184950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2185136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2185445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2185699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2186102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2186497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2186709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2187121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2187385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3sg8ieg
2022-11-23T01:51:49.2187656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3sg8ieg/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2187911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvrgdkjt
2022-11-23T01:51:49.2188179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvrgdkjt/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2188412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2188645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2188888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2189124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2189319Z ok (6.450s)
2022-11-23T01:51:49.2189339Z 
2022-11-23T01:51:49.2189618Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2189728Z Ran 1 test in 6.450s
2022-11-23T01:51:49.2189748Z 
2022-11-23T01:51:49.2189840Z OK
2022-11-23T01:51:49.2189859Z 
2022-11-23T01:51:49.2189986Z Generating XML reports...
2022-11-23T01:51:49.2190441Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013658.xml
2022-11-23T01:51:49.2190813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2190973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2191351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2191540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2191564Z 
2022-11-23T01:51:49.2191670Z Running tests...
2022-11-23T01:51:49.2191935Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2192247Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2192622Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2192842Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14475
2022-11-23T01:51:49.2193058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14476
2022-11-23T01:51:49.2193416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2193592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2193977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2194167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2194532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2194775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2195167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2195357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2195583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2195829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2196229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2196626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2196854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2197084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2197341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8r4gjrfp
2022-11-23T01:51:49.2197610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8r4gjrfp/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2197862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfhybcf4z
2022-11-23T01:51:49.2198113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfhybcf4z/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2198407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2198641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2198872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2199107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2199207Z ok (6.513s)
2022-11-23T01:51:49.2199228Z 
2022-11-23T01:51:49.2199500Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2199612Z Ran 1 test in 6.514s
2022-11-23T01:51:49.2199631Z 
2022-11-23T01:51:49.2199705Z OK
2022-11-23T01:51:49.2199743Z 
2022-11-23T01:51:49.2199848Z Generating XML reports...
2022-11-23T01:51:49.2200300Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013707.xml
2022-11-23T01:51:49.2200670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2200851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2201229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2201422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2201442Z 
2022-11-23T01:51:49.2201550Z Running tests...
2022-11-23T01:51:49.2201814Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2202107Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2202474Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2202692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14620
2022-11-23T01:51:49.2202914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14621
2022-11-23T01:51:49.2203283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2203456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2203885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2204083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2204457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2204615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2204988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2205181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2205425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2205829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2206076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2206475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2206703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2206913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2207166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpweidb9k5
2022-11-23T01:51:49.2207499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpweidb9k5/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2207751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaf1qsfew
2022-11-23T01:51:49.2208018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaf1qsfew/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2208255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2208490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2208720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2208950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2209033Z ok (6.455s)
2022-11-23T01:51:49.2209053Z 
2022-11-23T01:51:49.2209323Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2209440Z Ran 1 test in 6.455s
2022-11-23T01:51:49.2209459Z 
2022-11-23T01:51:49.2209551Z OK
2022-11-23T01:51:49.2209571Z 
2022-11-23T01:51:49.2209694Z Generating XML reports...
2022-11-23T01:51:49.2210147Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013716.xml
2022-11-23T01:51:49.2210521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2210698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2211060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2211250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2211269Z 
2022-11-23T01:51:49.2211377Z Running tests...
2022-11-23T01:51:49.2211637Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2211955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2212323Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2212593Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14765
2022-11-23T01:51:49.2212819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14766
2022-11-23T01:51:49.2213192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2213348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2213727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2213916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2214292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2214465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2214844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2215037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2215281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2215526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2215910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2216305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2216602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2216828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2217083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgk8s5oij
2022-11-23T01:51:49.2217353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgk8s5oij/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2217604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkfcbnjp
2022-11-23T01:51:49.2217874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkfcbnjp/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2218107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2218324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2218558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2218789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2218890Z ok (6.438s)
2022-11-23T01:51:49.2218910Z 
2022-11-23T01:51:49.2219183Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2219300Z Ran 1 test in 6.439s
2022-11-23T01:51:49.2219319Z 
2022-11-23T01:51:49.2219410Z OK
2022-11-23T01:51:49.2219430Z 
2022-11-23T01:51:49.2219552Z Generating XML reports...
2022-11-23T01:51:49.2219985Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013725.xml
2022-11-23T01:51:49.2220355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2220529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2220906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2221102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2221121Z 
2022-11-23T01:51:49.2221230Z Running tests...
2022-11-23T01:51:49.2221496Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2221857Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2222234Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2222435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14910
2022-11-23T01:51:49.2222650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14911
2022-11-23T01:51:49.2223024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2223202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2223580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2223769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2224135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2224310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2224668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2224857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2225102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2225406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2225810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2226210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2226440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2226668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2227300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmb6md0ko
2022-11-23T01:51:49.2227575Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmb6md0ko/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2227829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_h9viwr9
2022-11-23T01:51:49.2228101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_h9viwr9/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2228336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2228573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2228809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2229042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2229144Z ok (6.440s)
2022-11-23T01:51:49.2229165Z 
2022-11-23T01:51:49.2229450Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2229545Z Ran 1 test in 6.440s
2022-11-23T01:51:49.2229565Z 
2022-11-23T01:51:49.2229659Z OK
2022-11-23T01:51:49.2229679Z 
2022-11-23T01:51:49.2229801Z Generating XML reports...
2022-11-23T01:51:49.2230254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013734.xml
2022-11-23T01:51:49.2230633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2230810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2231281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2231487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2231508Z 
2022-11-23T01:51:49.2231598Z Running tests...
2022-11-23T01:51:49.2231871Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2232181Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2232552Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2232777Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15055
2022-11-23T01:51:49.2232992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15056
2022-11-23T01:51:49.2233373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2233548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2233929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2234104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2234473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2234648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2235116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2235306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2235549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2235799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2236201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2236599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2236811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2237041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2237304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5goekqb
2022-11-23T01:51:49.2237572Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5goekqb/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2237826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37woviql
2022-11-23T01:51:49.2238093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37woviql/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2238327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2238561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2238775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2239010Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2239114Z ok (6.413s)
2022-11-23T01:51:49.2239135Z 
2022-11-23T01:51:49.2239408Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2239520Z Ran 1 test in 6.413s
2022-11-23T01:51:49.2239540Z 
2022-11-23T01:51:49.2239632Z OK
2022-11-23T01:51:49.2239652Z 
2022-11-23T01:51:49.2239774Z Generating XML reports...
2022-11-23T01:51:49.2240283Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013743.xml
2022-11-23T01:51:49.2240676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2240836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2241214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2241404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2241429Z 
2022-11-23T01:51:49.2241536Z Running tests...
2022-11-23T01:51:49.2241803Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2242117Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2242486Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2242705Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15200
2022-11-23T01:51:49.2242908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15201
2022-11-23T01:51:49.2243282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2243459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2243839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2244096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2244469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2244647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2245026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2245215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2245441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2245685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2246086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2246488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2246718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2246949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2247205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlvdtf4w
2022-11-23T01:51:49.2247477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlvdtf4w/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2247731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkb6y23u
2022-11-23T01:51:49.2247978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkb6y23u/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2248215Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2248452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2248684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2248915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2249082Z ok (6.441s)
2022-11-23T01:51:49.2249105Z 
2022-11-23T01:51:49.2249385Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2249498Z Ran 1 test in 6.441s
2022-11-23T01:51:49.2249518Z 
2022-11-23T01:51:49.2249592Z OK
2022-11-23T01:51:49.2249610Z 
2022-11-23T01:51:49.2249735Z Generating XML reports...
2022-11-23T01:51:49.2250189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013752.xml
2022-11-23T01:51:49.2250561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2250744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2251127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2251318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2251342Z 
2022-11-23T01:51:49.2251449Z Running tests...
2022-11-23T01:51:49.2251714Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2252011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2252325Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2252547Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15345
2022-11-23T01:51:49.2252764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15346
2022-11-23T01:51:49.2253205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2253378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2253759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2253948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2254298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2254475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2254851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2255041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2255290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2255531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2255936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2256337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2256567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2256778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2257035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtef0pl3
2022-11-23T01:51:49.2257306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtef0pl3/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2257564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptei57tgq
2022-11-23T01:51:49.2257830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptei57tgq/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2258065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2258348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2258592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2258823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2258907Z ok (6.560s)
2022-11-23T01:51:49.2258927Z 
2022-11-23T01:51:49.2259212Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2259307Z Ran 1 test in 6.561s
2022-11-23T01:51:49.2259326Z 
2022-11-23T01:51:49.2259431Z OK
2022-11-23T01:51:49.2259451Z 
2022-11-23T01:51:49.2259577Z Generating XML reports...
2022-11-23T01:51:49.2260031Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013801.xml
2022-11-23T01:51:49.2260405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2260585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2260968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2261159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2261179Z 
2022-11-23T01:51:49.2261270Z Running tests...
2022-11-23T01:51:49.2261535Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2261845Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2262218Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2262437Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15490
2022-11-23T01:51:49.2262654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15491
2022-11-23T01:51:49.2263035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2263213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2263597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2263771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2264141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2264319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2264696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2264887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2265133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2265380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2265781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2266182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2266394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2266627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2267075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zu0cd38
2022-11-23T01:51:49.2267356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zu0cd38/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2267689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qyc1_xo
2022-11-23T01:51:49.2267966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qyc1_xo/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2268202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2268438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2268652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2268890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2268995Z ok (6.450s)
2022-11-23T01:51:49.2269016Z 
2022-11-23T01:51:49.2269296Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2269410Z Ran 1 test in 6.450s
2022-11-23T01:51:49.2269429Z 
2022-11-23T01:51:49.2269522Z OK
2022-11-23T01:51:49.2269542Z 
2022-11-23T01:51:49.2269669Z Generating XML reports...
2022-11-23T01:51:49.2270122Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013810.xml
2022-11-23T01:51:49.2270495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2270653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2271029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2271217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2271306Z 
2022-11-23T01:51:49.2271422Z Running tests...
2022-11-23T01:51:49.2271690Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2272003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2272274Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2273026Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s)
2022-11-23T01:51:49.2273046Z 
2022-11-23T01:51:49.2273308Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2273401Z Ran 1 test in 1.638s
2022-11-23T01:51:49.2273442Z 
2022-11-23T01:51:49.2273532Z OK (skipped=1)
2022-11-23T01:51:49.2273551Z 
2022-11-23T01:51:49.2273672Z Generating XML reports...
2022-11-23T01:51:49.2274120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013819.xml
2022-11-23T01:51:49.2274491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2274666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2275050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2275241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2275261Z 
2022-11-23T01:51:49.2275366Z Running tests...
2022-11-23T01:51:49.2275607Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2275915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2276173Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2276387Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15669
2022-11-23T01:51:49.2276605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15670
2022-11-23T01:51:49.2277027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2277212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2277593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2277765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2278132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2278310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2278685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2278874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2279122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2279365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2279765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2280159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2280372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2280656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2280912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ctcwg_c
2022-11-23T01:51:49.2281185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ctcwg_c/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2281442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_xp5g1u
2022-11-23T01:51:49.2281708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_xp5g1u/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2281810Z ok (6.546s)
2022-11-23T01:51:49.2281830Z 
2022-11-23T01:51:49.2282104Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2282198Z Ran 1 test in 6.546s
2022-11-23T01:51:49.2282235Z 
2022-11-23T01:51:49.2282311Z OK
2022-11-23T01:51:49.2282329Z 
2022-11-23T01:51:49.2282453Z Generating XML reports...
2022-11-23T01:51:49.2282907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013823.xml
2022-11-23T01:51:49.2285115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2285292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2285675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2285869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2285889Z 
2022-11-23T01:51:49.2285998Z Running tests...
2022-11-23T01:51:49.2286240Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2286555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2286827Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2287050Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15780
2022-11-23T01:51:49.2287265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15781
2022-11-23T01:51:49.2287695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2287879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2288253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2288428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2288786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2289020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2289397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2289589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2289834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2290083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2290483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2290879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2291109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2291321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2291637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsvi78sa
2022-11-23T01:51:49.2291908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsvi78sa/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2292160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmefn8br
2022-11-23T01:51:49.2292432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmefn8br/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2292666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2292902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2293306Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpbmefn8br'>
2022-11-23T01:51:49.2293452Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T01:51:49.2293848Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpdsvi78sa'>
2022-11-23T01:51:49.2294023Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T01:51:49.2294126Z ok (6.208s)
2022-11-23T01:51:49.2294147Z 
2022-11-23T01:51:49.2294410Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2294523Z Ran 1 test in 6.208s
2022-11-23T01:51:49.2294548Z 
2022-11-23T01:51:49.2294638Z OK
2022-11-23T01:51:49.2294658Z 
2022-11-23T01:51:49.2294783Z Generating XML reports...
2022-11-23T01:51:49.2295236Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013833.xml
2022-11-23T01:51:49.2295588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2295762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2296140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2296337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2296356Z 
2022-11-23T01:51:49.2296464Z Running tests...
2022-11-23T01:51:49.2296733Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2297100Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2297376Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2297578Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15895
2022-11-23T01:51:49.2297796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15896
2022-11-23T01:51:49.2298169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2298341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2298722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2298911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2299279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2299455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2299832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2300005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2300252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2300495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2300958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2301355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2301592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2301847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpud8a6goo
2022-11-23T01:51:49.2302116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpud8a6goo/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2302325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2302582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeo7nvgkr
2022-11-23T01:51:49.2302852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeo7nvgkr/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2303090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2303320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2303424Z ok (4.426s)
2022-11-23T01:51:49.2303445Z 
2022-11-23T01:51:49.2303721Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2303834Z Ran 1 test in 4.427s
2022-11-23T01:51:49.2303855Z 
2022-11-23T01:51:49.2303946Z OK
2022-11-23T01:51:49.2303965Z 
2022-11-23T01:51:49.2304070Z Generating XML reports...
2022-11-23T01:51:49.2304519Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013841.xml
2022-11-23T01:51:49.2304890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2305067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2305456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2305649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2305669Z 
2022-11-23T01:51:49.2305777Z Running tests...
2022-11-23T01:51:49.2306095Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2306404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2306670Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2307081Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16038
2022-11-23T01:51:49.2307307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16039
2022-11-23T01:51:49.2307686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2307869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2308247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2308436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2308806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2308963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2309345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2309535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2309780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2310024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2310528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2310924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2311158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2311386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2311624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtx_l9b7
2022-11-23T01:51:49.2311895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtx_l9b7/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2312149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplppwk2_o
2022-11-23T01:51:49.2312425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplppwk2_o/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2312659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2312894Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2312998Z ok (6.333s)
2022-11-23T01:51:49.2313018Z 
2022-11-23T01:51:49.2313291Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2313386Z Ran 1 test in 6.333s
2022-11-23T01:51:49.2313406Z 
2022-11-23T01:51:49.2313497Z OK
2022-11-23T01:51:49.2313517Z 
2022-11-23T01:51:49.2313640Z Generating XML reports...
2022-11-23T01:51:49.2314093Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013848.xml
2022-11-23T01:51:49.2314461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2314641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2315021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2315214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2315234Z 
2022-11-23T01:51:49.2315343Z Running tests...
2022-11-23T01:51:49.2315668Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2316001Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2316287Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2316505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16153
2022-11-23T01:51:49.2316723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16154
2022-11-23T01:51:49.2317095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2317278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2317661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2317835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2318205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2318380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2318752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2318943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2319187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2319498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2319902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2320300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2320514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2320743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2320980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2321220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2321616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2322012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2322255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:51:49.2322494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:51:49.2322887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.2323259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.2323522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7bzapk3
2022-11-23T01:51:49.2323793Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7bzapk3/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2324055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpotn05p_9
2022-11-23T01:51:49.2324323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpotn05p_9/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2324427Z ok (5.841s)
2022-11-23T01:51:49.2324447Z 
2022-11-23T01:51:49.2324828Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2324949Z Ran 1 test in 5.841s
2022-11-23T01:51:49.2324969Z 
2022-11-23T01:51:49.2325062Z OK
2022-11-23T01:51:49.2325082Z 
2022-11-23T01:51:49.2325188Z Generating XML reports...
2022-11-23T01:51:49.2325646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013857.xml
2022-11-23T01:51:49.2326018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2326195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2326579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2326772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2326791Z 
2022-11-23T01:51:49.2326900Z Running tests...
2022-11-23T01:51:49.2327165Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2327461Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2327745Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2327966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16276
2022-11-23T01:51:49.2328183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16277
2022-11-23T01:51:49.2328553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2328790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2329173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2329366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2329733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2329890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2330268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2330457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2330702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2330950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2331347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2331747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2331976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2332204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2332427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2332666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2333060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2333457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2333698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:51:49.2333992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:51:49.2334393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.2334781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.2335038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeew9t9l3
2022-11-23T01:51:49.2335293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeew9t9l3/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2335557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvodpavhl
2022-11-23T01:51:49.2335827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvodpavhl/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2335931Z ok (15.771s)
2022-11-23T01:51:49.2335952Z 
2022-11-23T01:51:49.2336224Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2336339Z Ran 1 test in 15.771s
2022-11-23T01:51:49.2336359Z 
2022-11-23T01:51:49.2336451Z OK
2022-11-23T01:51:49.2336470Z 
2022-11-23T01:51:49.2336594Z Generating XML reports...
2022-11-23T01:51:49.2337026Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013905.xml
2022-11-23T01:51:49.2337396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2337572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2338017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2338208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2338228Z 
2022-11-23T01:51:49.2338337Z Running tests...
2022-11-23T01:51:49.2338604Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2338917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2339225Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2339427Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16399
2022-11-23T01:51:49.2339642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16400
2022-11-23T01:51:49.2340015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2340195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2340577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2340764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2341134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2341309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2341668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2341858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2342104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2342356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2342760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2343203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2343442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2343669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2343927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5gsfgwz
2022-11-23T01:51:49.2344181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5gsfgwz/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2344433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlvxsrpt
2022-11-23T01:51:49.2344706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlvxsrpt/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2344808Z ok (6.552s)
2022-11-23T01:51:49.2344829Z 
2022-11-23T01:51:49.2345097Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2345210Z Ran 1 test in 6.552s
2022-11-23T01:51:49.2345230Z 
2022-11-23T01:51:49.2345324Z OK
2022-11-23T01:51:49.2345344Z 
2022-11-23T01:51:49.2345472Z Generating XML reports...
2022-11-23T01:51:49.2345923Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013924.xml
2022-11-23T01:51:49.2346276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2346453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2347011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2347307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2347327Z 
2022-11-23T01:51:49.2347435Z Running tests...
2022-11-23T01:51:49.2347710Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2348028Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2348319Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2348520Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16514
2022-11-23T01:51:49.2348737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16515
2022-11-23T01:51:49.2349110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2349285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2349668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2349856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2350227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2350403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2350778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2350950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2351194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2351434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2351844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2352239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2352536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2352775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2353034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb21bffhn
2022-11-23T01:51:49.2353305Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb21bffhn/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2353539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bx8pyi7
2022-11-23T01:51:49.2353804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bx8pyi7/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2353911Z ok (6.560s)
2022-11-23T01:51:49.2353931Z 
2022-11-23T01:51:49.2354202Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2354314Z Ran 1 test in 6.560s
2022-11-23T01:51:49.2354334Z 
2022-11-23T01:51:49.2354427Z OK
2022-11-23T01:51:49.2354447Z 
2022-11-23T01:51:49.2354569Z Generating XML reports...
2022-11-23T01:51:49.2355021Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013933.xml
2022-11-23T01:51:49.2355375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2355553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2355934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2356126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2356198Z 
2022-11-23T01:51:49.2356310Z Running tests...
2022-11-23T01:51:49.2356576Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2356888Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2357145Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2357364Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16629
2022-11-23T01:51:49.2357563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16630
2022-11-23T01:51:49.2357935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2358109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2358487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2358682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2359048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2359222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2359598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2359767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2360012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2360256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2360655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2361057Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2361287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2361512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2361817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5rlvr_7z
2022-11-23T01:51:49.2362094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5rlvr_7z/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2362329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3a9o9t0e
2022-11-23T01:51:49.2362594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3a9o9t0e/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2362695Z ok (6.262s)
2022-11-23T01:51:49.2362715Z 
2022-11-23T01:51:49.2362984Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2363102Z Ran 1 test in 6.262s
2022-11-23T01:51:49.2363122Z 
2022-11-23T01:51:49.2363216Z OK
2022-11-23T01:51:49.2363235Z 
2022-11-23T01:51:49.2363359Z Generating XML reports...
2022-11-23T01:51:49.2363809Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013942.xml
2022-11-23T01:51:49.2364167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2364342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2364721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2364915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2364934Z 
2022-11-23T01:51:49.2365042Z Running tests...
2022-11-23T01:51:49.2365304Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2365683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2365946Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2366163Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16740
2022-11-23T01:51:49.2366364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16741
2022-11-23T01:51:49.2366732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2366907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2367283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2367475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2367849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2368025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2368402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2368577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2368823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2369066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2369465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2369860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2370095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2370324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2370582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeq0ve8eg
2022-11-23T01:51:49.2370900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeq0ve8eg/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2371143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpana3fu
2022-11-23T01:51:49.2371413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpana3fu/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2372192Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:51:49.2372976Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:51:49.2373216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2373452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2373605Z ok (6.252s)
2022-11-23T01:51:49.2373626Z 
2022-11-23T01:51:49.2373897Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2374008Z Ran 1 test in 6.252s
2022-11-23T01:51:49.2374028Z 
2022-11-23T01:51:49.2374119Z OK
2022-11-23T01:51:49.2374139Z 
2022-11-23T01:51:49.2374266Z Generating XML reports...
2022-11-23T01:51:49.2374725Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013951.xml
2022-11-23T01:51:49.2375077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2375256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2375637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2375830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2375854Z 
2022-11-23T01:51:49.2375962Z Running tests...
2022-11-23T01:51:49.2376231Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2376543Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2376828Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2377576Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s)
2022-11-23T01:51:49.2377597Z 
2022-11-23T01:51:49.2377857Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2377952Z Ran 1 test in 1.645s
2022-11-23T01:51:49.2377975Z 
2022-11-23T01:51:49.2378084Z OK (skipped=1)
2022-11-23T01:51:49.2378104Z 
2022-11-23T01:51:49.2378226Z Generating XML reports...
2022-11-23T01:51:49.2378674Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014000.xml
2022-11-23T01:51:49.2379109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2379293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2379679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2379869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2379889Z 
2022-11-23T01:51:49.2379999Z Running tests...
2022-11-23T01:51:49.2380243Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2380558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2380846Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2381590Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.606s)
2022-11-23T01:51:49.2381612Z 
2022-11-23T01:51:49.2381872Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2381984Z Ran 1 test in 1.606s
2022-11-23T01:51:49.2382003Z 
2022-11-23T01:51:49.2382110Z OK (skipped=1)
2022-11-23T01:51:49.2382129Z 
2022-11-23T01:51:49.2382251Z Generating XML reports...
2022-11-23T01:51:49.2382701Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014004.xml
2022-11-23T01:51:49.2383118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2383297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2383677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2383870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2383889Z 
2022-11-23T01:51:49.2383996Z Running tests...
2022-11-23T01:51:49.2384257Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2384570Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2384847Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2385065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16923
2022-11-23T01:51:49.2385270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16924
2022-11-23T01:51:49.2385643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2385819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2386202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2386394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2386938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2387133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2387521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2387698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2387994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2388236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2388723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2389170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2389401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2389630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2389887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4s1psij
2022-11-23T01:51:49.2390162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4s1psij/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2390397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fzzhydw
2022-11-23T01:51:49.2390668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fzzhydw/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2391017Z STAGE:2022-11-23 01:40:14 16923:16923 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2391340Z STAGE:2022-11-23 01:40:14 16924:16924 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2391581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2391815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2392367Z STAGE:2022-11-23 01:40:14 16923:16923 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:40:14 16924:16924 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2392457Z 
2022-11-23T01:51:49.2392821Z STAGE:2022-11-23 01:40:14 16924:16924 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2393169Z STAGE:2022-11-23 01:40:14 16923:16923 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2393952Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:51:49.2394729Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:51:49.2395068Z STAGE:2022-11-23 01:40:14 16923:16923 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2395390Z STAGE:2022-11-23 01:40:14 16924:16924 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2395704Z STAGE:2022-11-23 01:40:14 16923:16923 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2396033Z STAGE:2022-11-23 01:40:14 16924:16924 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2396381Z STAGE:2022-11-23 01:40:14 16923:16923 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2396731Z STAGE:2022-11-23 01:40:14 16924:16924 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2396835Z ok (6.751s)
2022-11-23T01:51:49.2396855Z 
2022-11-23T01:51:49.2397119Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2397281Z Ran 1 test in 6.751s
2022-11-23T01:51:49.2397303Z 
2022-11-23T01:51:49.2397398Z OK
2022-11-23T01:51:49.2397417Z 
2022-11-23T01:51:49.2397543Z Generating XML reports...
2022-11-23T01:51:49.2397980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014008.xml
2022-11-23T01:51:49.2398352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2398528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2398910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2399105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2399125Z 
2022-11-23T01:51:49.2399233Z Running tests...
2022-11-23T01:51:49.2399501Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2399823Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2400090Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2400290Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17042
2022-11-23T01:51:49.2400510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17043
2022-11-23T01:51:49.2400882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2401116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2401496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2401687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2402056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2402228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2402582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2402770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2403012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2403255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2403660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2404057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2404288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2404521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2404779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphipzkcu1
2022-11-23T01:51:49.2405031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphipzkcu1/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2405283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3frph7oi
2022-11-23T01:51:49.2405551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3frph7oi/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2405659Z ok (5.753s)
2022-11-23T01:51:49.2405679Z 
2022-11-23T01:51:49.2405948Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2406060Z Ran 1 test in 5.753s
2022-11-23T01:51:49.2406079Z 
2022-11-23T01:51:49.2406172Z OK
2022-11-23T01:51:49.2406191Z 
2022-11-23T01:51:49.2406366Z Generating XML reports...
2022-11-23T01:51:49.2406810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014017.xml
2022-11-23T01:51:49.2407182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2407358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2407736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2407927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2407951Z 
2022-11-23T01:51:49.2408059Z Running tests...
2022-11-23T01:51:49.2408321Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2408631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2408914Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2409648Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.653s)
2022-11-23T01:51:49.2409688Z 
2022-11-23T01:51:49.2409930Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2410041Z Ran 1 test in 1.653s
2022-11-23T01:51:49.2410111Z 
2022-11-23T01:51:49.2410224Z OK (skipped=1)
2022-11-23T01:51:49.2410243Z 
2022-11-23T01:51:49.2410366Z Generating XML reports...
2022-11-23T01:51:49.2410820Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014026.xml
2022-11-23T01:51:49.2411189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2411365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2411743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2411933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2411954Z 
2022-11-23T01:51:49.2412043Z Running tests...
2022-11-23T01:51:49.2412303Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2412610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2412895Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2413116Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17187
2022-11-23T01:51:49.2413335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17188
2022-11-23T01:51:49.2413706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2413877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2414232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2414421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2414784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2414963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2415339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2415530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2415826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2416078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2416481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2416862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2417093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2417324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2417581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjrcj2vwf
2022-11-23T01:51:49.2417857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjrcj2vwf/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2418108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9dmqrb33
2022-11-23T01:51:49.2418377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9dmqrb33/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2419288Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T01:51:49.2419458Z   warnings.warn(
2022-11-23T01:51:49.2420376Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T01:51:49.2420488Z   warnings.warn(
2022-11-23T01:51:49.2420711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2420944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2421047Z ok (6.263s)
2022-11-23T01:51:49.2421067Z 
2022-11-23T01:51:49.2421334Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2421447Z Ran 1 test in 6.264s
2022-11-23T01:51:49.2421471Z 
2022-11-23T01:51:49.2421565Z OK
2022-11-23T01:51:49.2421585Z 
2022-11-23T01:51:49.2421709Z Generating XML reports...
2022-11-23T01:51:49.2422158Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014030.xml
2022-11-23T01:51:49.2422513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2422692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2423072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2423263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2423283Z 
2022-11-23T01:51:49.2423390Z Running tests...
2022-11-23T01:51:49.2423652Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2423965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2424246Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2425046Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.608s)
2022-11-23T01:51:49.2425070Z 
2022-11-23T01:51:49.2425321Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2425434Z Ran 1 test in 1.608s
2022-11-23T01:51:49.2425453Z 
2022-11-23T01:51:49.2425561Z OK (skipped=1)
2022-11-23T01:51:49.2425580Z 
2022-11-23T01:51:49.2425703Z Generating XML reports...
2022-11-23T01:51:49.2426155Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014039.xml
2022-11-23T01:51:49.2426529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2426704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2427349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2427540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2427563Z 
2022-11-23T01:51:49.2427652Z Running tests...
2022-11-23T01:51:49.2427916Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2428226Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2428493Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2428710Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17336
2022-11-23T01:51:49.2429023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17337
2022-11-23T01:51:49.2429396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2429572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2429937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2430131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2430497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2430671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2431047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2431240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2431485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2431734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2432142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2432521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2432750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2432977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2433236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpph2ntby8
2022-11-23T01:51:49.2433512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpph2ntby8/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2433762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqby19nys
2022-11-23T01:51:49.2434030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqby19nys/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2434437Z STAGE:2022-11-23 01:40:48 17337:17337 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2434779Z STAGE:2022-11-23 01:40:48 17336:17336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2434999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2435232Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T01:51:49.2435774Z STAGE:2022-11-23 01:40:48 17336:17336 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:40:48 17337:17337 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2435800Z 
2022-11-23T01:51:49.2436152Z STAGE:2022-11-23 01:40:48 17336:17336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2436498Z STAGE:2022-11-23 01:40:48 17337:17337 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2436827Z STAGE:2022-11-23 01:40:48 17336:17336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2437158Z STAGE:2022-11-23 01:40:49 17336:17336 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2437502Z STAGE:2022-11-23 01:40:49 17336:17336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2437605Z ok (7.141s)
2022-11-23T01:51:49.2437625Z 
2022-11-23T01:51:49.2437872Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2437986Z Ran 1 test in 7.141s
2022-11-23T01:51:49.2438006Z 
2022-11-23T01:51:49.2438155Z OK
2022-11-23T01:51:49.2438174Z 
2022-11-23T01:51:49.2438299Z Generating XML reports...
2022-11-23T01:51:49.2438754Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014043.xml
2022-11-23T01:51:49.2439131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2439309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2439692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2439885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2439905Z 
2022-11-23T01:51:49.2439995Z Running tests...
2022-11-23T01:51:49.2440254Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2440567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2440839Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2441057Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17455
2022-11-23T01:51:49.2441274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17456
2022-11-23T01:51:49.2441648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2441822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2442182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2442374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2442741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2442919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2443298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2443488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2443797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2444050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2444455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2444831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2445064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2445296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2445553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29nj52_x
2022-11-23T01:51:49.2445821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29nj52_x/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2446078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcquwjaeh
2022-11-23T01:51:49.2446347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcquwjaeh/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2446450Z ok (5.721s)
2022-11-23T01:51:49.2446470Z 
2022-11-23T01:51:49.2446741Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2446837Z Ran 1 test in 5.721s
2022-11-23T01:51:49.2446856Z 
2022-11-23T01:51:49.2446950Z OK
2022-11-23T01:51:49.2446969Z 
2022-11-23T01:51:49.2447093Z Generating XML reports...
2022-11-23T01:51:49.2447541Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014052.xml
2022-11-23T01:51:49.2447977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2448152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2448537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2448731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2448750Z 
2022-11-23T01:51:49.2448840Z Running tests...
2022-11-23T01:51:49.2449101Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2449413Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2449687Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2449911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17566
2022-11-23T01:51:49.2450130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17567
2022-11-23T01:51:49.2450500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2450682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2451066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2451239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2451609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2451783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2452157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2452352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2452597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2452894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2453306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2453682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2453912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2454143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2454401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0cmxn51b
2022-11-23T01:51:49.2454669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0cmxn51b/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2454920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjucmmvu3
2022-11-23T01:51:49.2455186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjucmmvu3/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2455292Z ok (5.757s)
2022-11-23T01:51:49.2455312Z 
2022-11-23T01:51:49.2455581Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2455675Z Ran 1 test in 5.758s
2022-11-23T01:51:49.2455695Z 
2022-11-23T01:51:49.2455790Z OK
2022-11-23T01:51:49.2455809Z 
2022-11-23T01:51:49.2455931Z Generating XML reports...
2022-11-23T01:51:49.2456382Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014101.xml
2022-11-23T01:51:49.2456818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2456993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2457376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2457572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2457592Z 
2022-11-23T01:51:49.2457700Z Running tests...
2022-11-23T01:51:49.2457948Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2458260Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2458534Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2459282Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.616s)
2022-11-23T01:51:49.2459307Z 
2022-11-23T01:51:49.2459571Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2459683Z Ran 1 test in 1.617s
2022-11-23T01:51:49.2459706Z 
2022-11-23T01:51:49.2459814Z OK (skipped=1)
2022-11-23T01:51:49.2459833Z 
2022-11-23T01:51:49.2459957Z Generating XML reports...
2022-11-23T01:51:49.2460404Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014109.xml
2022-11-23T01:51:49.2460755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2460931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2461307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2461500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2461519Z 
2022-11-23T01:51:49.2461627Z Running tests...
2022-11-23T01:51:49.2461889Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2462257Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2462526Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2463275Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-11-23T01:51:49.2463296Z 
2022-11-23T01:51:49.2463557Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2463656Z Ran 1 test in 1.642s
2022-11-23T01:51:49.2463675Z 
2022-11-23T01:51:49.2463782Z OK (skipped=1)
2022-11-23T01:51:49.2463801Z 
2022-11-23T01:51:49.2463924Z Generating XML reports...
2022-11-23T01:51:49.2464377Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014113.xml
2022-11-23T01:51:49.2464749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2464924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2465302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2465492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2465512Z 
2022-11-23T01:51:49.2465619Z Running tests...
2022-11-23T01:51:49.2465862Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2466288Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2466579Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2467575Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2022-11-23T01:51:49.2467599Z 
2022-11-23T01:51:49.2467864Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2467976Z Ran 1 test in 1.621s
2022-11-23T01:51:49.2467996Z 
2022-11-23T01:51:49.2468105Z OK (skipped=1)
2022-11-23T01:51:49.2468124Z 
2022-11-23T01:51:49.2468248Z Generating XML reports...
2022-11-23T01:51:49.2468704Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014117.xml
2022-11-23T01:51:49.2469055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2469231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2469609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2469801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2469820Z 
2022-11-23T01:51:49.2469928Z Running tests...
2022-11-23T01:51:49.2470185Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2470494Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2470790Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2471015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17779
2022-11-23T01:51:49.2471215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17780
2022-11-23T01:51:49.2471672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2471860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2472244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2472433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2472801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2472977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2473361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2473532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2473776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2474024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2474426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2474823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2475053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2475279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2475611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpce8lriyi
2022-11-23T01:51:49.2475882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpce8lriyi/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2476116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplevslk3q
2022-11-23T01:51:49.2476389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplevslk3q/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2476491Z ok (6.353s)
2022-11-23T01:51:49.2476511Z 
2022-11-23T01:51:49.2476784Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2476894Z Ran 1 test in 6.353s
2022-11-23T01:51:49.2476913Z 
2022-11-23T01:51:49.2477002Z OK
2022-11-23T01:51:49.2477022Z 
2022-11-23T01:51:49.2477146Z Generating XML reports...
2022-11-23T01:51:49.2477598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014121.xml
2022-11-23T01:51:49.2477978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2478135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2478516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2478711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2478731Z 
2022-11-23T01:51:49.2478839Z Running tests...
2022-11-23T01:51:49.2479098Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2479410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2479677Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2479896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17894
2022-11-23T01:51:49.2480102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17895
2022-11-23T01:51:49.2480475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2480652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2481084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2481282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2481659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2481834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2482206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2482403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2482630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2482877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2483284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2483681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2483911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2484292Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T01:51:49.2484548Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T01:51:49.2484836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2485199Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T01:51:49.2485451Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T01:51:49.2485711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg0u9pbmo
2022-11-23T01:51:49.2485982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg0u9pbmo/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2486237Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ny41x4r
2022-11-23T01:51:49.2486502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ny41x4r/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2486602Z ok (5.854s)
2022-11-23T01:51:49.2486622Z 
2022-11-23T01:51:49.2486893Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2487011Z Ran 1 test in 5.854s
2022-11-23T01:51:49.2487032Z 
2022-11-23T01:51:49.2487107Z OK
2022-11-23T01:51:49.2487126Z 
2022-11-23T01:51:49.2487249Z Generating XML reports...
2022-11-23T01:51:49.2487696Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014130.xml
2022-11-23T01:51:49.2488070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2488247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2488627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2488820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2488840Z 
2022-11-23T01:51:49.2488985Z Running tests...
2022-11-23T01:51:49.2489236Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2489556Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2489818Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2490036Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18005
2022-11-23T01:51:49.2490304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18006
2022-11-23T01:51:49.2490685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2490861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2491234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2491423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2491774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2491958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2492334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2492524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2492770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2493015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2493415Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2493799Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2494029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2494299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2494541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2494788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2495191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2495578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2495681Z ok (4.325s)
2022-11-23T01:51:49.2495701Z 
2022-11-23T01:51:49.2495963Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2496075Z Ran 1 test in 4.325s
2022-11-23T01:51:49.2496095Z 
2022-11-23T01:51:49.2496174Z OK
2022-11-23T01:51:49.2496212Z 
2022-11-23T01:51:49.2496316Z Generating XML reports...
2022-11-23T01:51:49.2496772Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014139.xml
2022-11-23T01:51:49.2497142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2497322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2497702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2497890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2497910Z 
2022-11-23T01:51:49.2498017Z Running tests...
2022-11-23T01:51:49.2498280Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2498575Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2498834Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2499050Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18120
2022-11-23T01:51:49.2499267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18121
2022-11-23T01:51:49.2499682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2499865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2500246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2500439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2500787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2500967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2501339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2501526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2501773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2502016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2502415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2502811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2503036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2503302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2503545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2503783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2504189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2504582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2504684Z ok (4.221s)
2022-11-23T01:51:49.2504703Z 
2022-11-23T01:51:49.2504963Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2505077Z Ran 1 test in 4.221s
2022-11-23T01:51:49.2505096Z 
2022-11-23T01:51:49.2505190Z OK
2022-11-23T01:51:49.2505209Z 
2022-11-23T01:51:49.2505315Z Generating XML reports...
2022-11-23T01:51:49.2505774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014146.xml
2022-11-23T01:51:49.2506149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2506329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2506711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2507137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2507160Z 
2022-11-23T01:51:49.2507271Z Running tests...
2022-11-23T01:51:49.2507544Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2507841Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2508121Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2508878Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.600s)
2022-11-23T01:51:49.2508990Z 
2022-11-23T01:51:49.2509268Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2509381Z Ran 1 test in 1.600s
2022-11-23T01:51:49.2509400Z 
2022-11-23T01:51:49.2509509Z OK (skipped=1)
2022-11-23T01:51:49.2509529Z 
2022-11-23T01:51:49.2509652Z Generating XML reports...
2022-11-23T01:51:49.2510100Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014152.xml
2022-11-23T01:51:49.2510468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2510645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2511008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2511199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2511219Z 
2022-11-23T01:51:49.2511330Z Running tests...
2022-11-23T01:51:49.2511590Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2511901Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2512174Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2512915Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.626s)
2022-11-23T01:51:49.2513011Z 
2022-11-23T01:51:49.2513282Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2513394Z Ran 1 test in 1.626s
2022-11-23T01:51:49.2513414Z 
2022-11-23T01:51:49.2513522Z OK (skipped=1)
2022-11-23T01:51:49.2513542Z 
2022-11-23T01:51:49.2513650Z Generating XML reports...
2022-11-23T01:51:49.2514101Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014156.xml
2022-11-23T01:51:49.2514472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2514649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2515028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2515224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2515243Z 
2022-11-23T01:51:49.2515350Z Running tests...
2022-11-23T01:51:49.2515613Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2515906Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2516183Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2516403Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18303
2022-11-23T01:51:49.2516617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18304
2022-11-23T01:51:49.2516988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2517164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2517544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2517740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2518111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2518321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2518711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2518899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2519143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2519386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2519784Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2520188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2520422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2520654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2520738Z ok (4.252s)
2022-11-23T01:51:49.2520758Z 
2022-11-23T01:51:49.2521024Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2521137Z Ran 1 test in 4.252s
2022-11-23T01:51:49.2521157Z 
2022-11-23T01:51:49.2521248Z OK
2022-11-23T01:51:49.2521267Z 
2022-11-23T01:51:49.2521391Z Generating XML reports...
2022-11-23T01:51:49.2521841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014201.xml
2022-11-23T01:51:49.2522215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2522447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2522811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2523008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2523027Z 
2022-11-23T01:51:49.2523135Z Running tests...
2022-11-23T01:51:49.2523400Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2523711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2523952Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2524171Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18412
2022-11-23T01:51:49.2524389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18413
2022-11-23T01:51:49.2524766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2524923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2525304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2525495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2525862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2526036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2526408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2526598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2526845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2527070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2527519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2527927Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2528156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2528385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2528722Z STAGE:2022-11-23 01:42:11 18413:18413 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2529045Z STAGE:2022-11-23 01:42:11 18412:18412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2529385Z STAGE:2022-11-23 01:42:11 18413:18413 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2529729Z STAGE:2022-11-23 01:42:11 18413:18413 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2530047Z STAGE:2022-11-23 01:42:11 18412:18412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2530389Z STAGE:2022-11-23 01:42:11 18412:18412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2530709Z STAGE:2022-11-23 01:42:11 18413:18413 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2531027Z STAGE:2022-11-23 01:42:11 18412:18412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2531357Z STAGE:2022-11-23 01:42:11 18412:18412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2531700Z STAGE:2022-11-23 01:42:11 18412:18412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2532088Z STAGE:2022-11-23 01:42:11 18413:18413 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2532430Z STAGE:2022-11-23 01:42:11 18413:18413 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2532532Z ok (4.130s)
2022-11-23T01:51:49.2532556Z 
2022-11-23T01:51:49.2532802Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2532914Z Ran 1 test in 4.131s
2022-11-23T01:51:49.2532933Z 
2022-11-23T01:51:49.2533026Z OK
2022-11-23T01:51:49.2533045Z 
2022-11-23T01:51:49.2533169Z Generating XML reports...
2022-11-23T01:51:49.2533620Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014207.xml
2022-11-23T01:51:49.2533989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2534170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2534551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2534725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2534764Z 
2022-11-23T01:51:49.2534860Z Running tests...
2022-11-23T01:51:49.2535127Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2535440Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2535690Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2535907Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18525
2022-11-23T01:51:49.2536125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18526
2022-11-23T01:51:49.2536497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2536673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2537035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2537277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2537654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2537830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2538205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2538395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2538647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2538895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2539278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2539678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2539910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2540136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2540241Z ok (4.342s)
2022-11-23T01:51:49.2540261Z 
2022-11-23T01:51:49.2540526Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2540641Z Ran 1 test in 4.343s
2022-11-23T01:51:49.2540660Z 
2022-11-23T01:51:49.2540753Z OK
2022-11-23T01:51:49.2540773Z 
2022-11-23T01:51:49.2540950Z Generating XML reports...
2022-11-23T01:51:49.2541387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014214.xml
2022-11-23T01:51:49.2541758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2541937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2542315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2542503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2542523Z 
2022-11-23T01:51:49.2542631Z Running tests...
2022-11-23T01:51:49.2542894Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2543207Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2543463Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T01:51:49.2543487Z 
2022-11-23T01:51:49.2543732Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2543842Z Ran 1 test in 0.002s
2022-11-23T01:51:49.2543861Z 
2022-11-23T01:51:49.2543966Z OK (skipped=1)
2022-11-23T01:51:49.2543986Z 
2022-11-23T01:51:49.2544113Z Generating XML reports...
2022-11-23T01:51:49.2544562Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014221.xml
2022-11-23T01:51:49.2544934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2545109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2545486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2545659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2545702Z 
2022-11-23T01:51:49.2545793Z Running tests...
2022-11-23T01:51:49.2546057Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2546368Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2546675Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2547139Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18667
2022-11-23T01:51:49.2547365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18668
2022-11-23T01:51:49.2547747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2547921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2548282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2548481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2548848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2549026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2549403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2549593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2549835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2550078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2550460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2550962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2551194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2551428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2551668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2551907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2552301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2552692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2553031Z STAGE:2022-11-23 01:42:27 18668:18668 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2553354Z STAGE:2022-11-23 01:42:27 18667:18667 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2553672Z STAGE:2022-11-23 01:42:27 18668:18668 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2554024Z STAGE:2022-11-23 01:42:27 18668:18668 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2554358Z STAGE:2022-11-23 01:42:27 18667:18667 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2554701Z STAGE:2022-11-23 01:42:27 18667:18667 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2555022Z STAGE:2022-11-23 01:42:27 18668:18668 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2555337Z STAGE:2022-11-23 01:42:27 18667:18667 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2555670Z STAGE:2022-11-23 01:42:27 18667:18667 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2556017Z STAGE:2022-11-23 01:42:27 18667:18667 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2556351Z STAGE:2022-11-23 01:42:27 18668:18668 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2556758Z STAGE:2022-11-23 01:42:27 18668:18668 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2556872Z ok (4.456s)
2022-11-23T01:51:49.2556893Z 
2022-11-23T01:51:49.2557160Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2557273Z Ran 1 test in 4.457s
2022-11-23T01:51:49.2557294Z 
2022-11-23T01:51:49.2557386Z OK
2022-11-23T01:51:49.2557405Z 
2022-11-23T01:51:49.2557530Z Generating XML reports...
2022-11-23T01:51:49.2557983Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014223.xml
2022-11-23T01:51:49.2558359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2558517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2558903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2559095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2559115Z 
2022-11-23T01:51:49.2559225Z Running tests...
2022-11-23T01:51:49.2559487Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2559800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2560053Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2560270Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18786
2022-11-23T01:51:49.2560529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18787
2022-11-23T01:51:49.2560905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2561081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2561462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2561652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2562016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2562191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2562566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2562759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2562987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2563233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2563641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2564036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2564264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2564492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2564652Z skip: Skipped due to small world size. (4.262s)
2022-11-23T01:51:49.2564672Z 
2022-11-23T01:51:49.2564949Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2565061Z Ran 1 test in 4.263s
2022-11-23T01:51:49.2565081Z 
2022-11-23T01:51:49.2565170Z OK (skipped=1)
2022-11-23T01:51:49.2565189Z 
2022-11-23T01:51:49.2565313Z Generating XML reports...
2022-11-23T01:51:49.2565813Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014231.xml
2022-11-23T01:51:49.2566198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2566373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2566751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2566941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2566962Z 
2022-11-23T01:51:49.2567071Z Running tests...
2022-11-23T01:51:49.2567319Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2567634Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2567887Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2568105Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18895
2022-11-23T01:51:49.2568324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18896
2022-11-23T01:51:49.2568693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2568870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2569246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2569437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2569847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2570021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2570391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2570586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2570828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2571072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2571469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2571866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2572102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2572311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2572414Z ok (4.148s)
2022-11-23T01:51:49.2572434Z 
2022-11-23T01:51:49.2572705Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2572816Z Ran 1 test in 4.148s
2022-11-23T01:51:49.2572835Z 
2022-11-23T01:51:49.2572928Z OK
2022-11-23T01:51:49.2572947Z 
2022-11-23T01:51:49.2573073Z Generating XML reports...
2022-11-23T01:51:49.2573522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014237.xml
2022-11-23T01:51:49.2573887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2574047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2574429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2574620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2574640Z 
2022-11-23T01:51:49.2574747Z Running tests...
2022-11-23T01:51:49.2575073Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2575397Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2575663Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2576409Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-11-23T01:51:49.2576434Z 
2022-11-23T01:51:49.2576696Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2576809Z Ran 1 test in 1.642s
2022-11-23T01:51:49.2576829Z 
2022-11-23T01:51:49.2576918Z OK (skipped=1)
2022-11-23T01:51:49.2576937Z 
2022-11-23T01:51:49.2577060Z Generating XML reports...
2022-11-23T01:51:49.2577515Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014244.xml
2022-11-23T01:51:49.2577888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2578066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2578444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2578632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2578705Z 
2022-11-23T01:51:49.2578820Z Running tests...
2022-11-23T01:51:49.2579072Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2579385Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2579633Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2579856Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19038
2022-11-23T01:51:49.2580074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19039
2022-11-23T01:51:49.2580440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2580615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2580993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2581187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2581539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2581713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2582093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2582282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2582525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2582767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2583168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2583563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2583802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2584012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2584304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2584552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2584950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2585337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2585439Z ok (4.368s)
2022-11-23T01:51:49.2585460Z 
2022-11-23T01:51:49.2585721Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2585836Z Ran 1 test in 4.368s
2022-11-23T01:51:49.2585856Z 
2022-11-23T01:51:49.2585930Z OK
2022-11-23T01:51:49.2585967Z 
2022-11-23T01:51:49.2586073Z Generating XML reports...
2022-11-23T01:51:49.2586529Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014248.xml
2022-11-23T01:51:49.2587136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2587319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2587711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2587902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2587922Z 
2022-11-23T01:51:49.2588032Z Running tests...
2022-11-23T01:51:49.2588291Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2588686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2588967Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2589189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19153
2022-11-23T01:51:49.2589411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19154
2022-11-23T01:51:49.2589783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2589959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2590337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2590528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2590883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2591062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2591438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2591631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2591876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2592124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2592525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2592921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2593154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2593366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2593468Z ok (4.248s)
2022-11-23T01:51:49.2593489Z 
2022-11-23T01:51:49.2593818Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2593939Z Ran 1 test in 4.248s
2022-11-23T01:51:49.2593960Z 
2022-11-23T01:51:49.2594054Z OK
2022-11-23T01:51:49.2594074Z 
2022-11-23T01:51:49.2594197Z Generating XML reports...
2022-11-23T01:51:49.2594649Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014255.xml
2022-11-23T01:51:49.2595022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2595196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2595563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2595752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2595772Z 
2022-11-23T01:51:49.2595879Z Running tests...
2022-11-23T01:51:49.2596148Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2596461Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2596707Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2596929Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19262
2022-11-23T01:51:49.2597151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19263
2022-11-23T01:51:49.2597506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2597743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2598123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2598311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2598681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2598857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2599233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2599419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2599645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2599890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2600294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2600690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2600923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2601150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2601253Z ok (4.462s)
2022-11-23T01:51:49.2601273Z 
2022-11-23T01:51:49.2601538Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2601649Z Ran 1 test in 4.462s
2022-11-23T01:51:49.2601669Z 
2022-11-23T01:51:49.2601743Z OK
2022-11-23T01:51:49.2601762Z 
2022-11-23T01:51:49.2601886Z Generating XML reports...
2022-11-23T01:51:49.2602340Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014302.xml
2022-11-23T01:51:49.2602717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2602893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2603326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2603527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2603547Z 
2022-11-23T01:51:49.2603655Z Running tests...
2022-11-23T01:51:49.2603921Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2604216Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2604486Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2604710Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19371
2022-11-23T01:51:49.2604924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19372
2022-11-23T01:51:49.2605296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2605473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2605855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2606044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2606391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2606569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2606947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2607196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2607442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2607689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2608095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2608490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2608720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2608929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2609170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2609412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2609809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2610207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2610311Z ok (4.340s)
2022-11-23T01:51:49.2610330Z 
2022-11-23T01:51:49.2610593Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2610704Z Ran 1 test in 4.340s
2022-11-23T01:51:49.2610724Z 
2022-11-23T01:51:49.2610818Z OK
2022-11-23T01:51:49.2610838Z 
2022-11-23T01:51:49.2610942Z Generating XML reports...
2022-11-23T01:51:49.2611395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014309.xml
2022-11-23T01:51:49.2611776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2611954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2612433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2612631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2612651Z 
2022-11-23T01:51:49.2612752Z Running tests...
2022-11-23T01:51:49.2623758Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2624099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2624367Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2624591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19486
2022-11-23T01:51:49.2624822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19487
2022-11-23T01:51:49.2625206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2625384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2625773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2625952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2626321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2626496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2627177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2627538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2627785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2628031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2628458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2628859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2629074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2629302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2629543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2629788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2630188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2630588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2630691Z ok (4.334s)
2022-11-23T01:51:49.2630714Z 
2022-11-23T01:51:49.2630980Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2631076Z Ran 1 test in 4.334s
2022-11-23T01:51:49.2631112Z 
2022-11-23T01:51:49.2631185Z OK
2022-11-23T01:51:49.2631205Z 
2022-11-23T01:51:49.2631327Z Generating XML reports...
2022-11-23T01:51:49.2631783Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014316.xml
2022-11-23T01:51:49.2632156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2632338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2632723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2632989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2633012Z 
2022-11-23T01:51:49.2633129Z Running tests...
2022-11-23T01:51:49.2633382Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2633695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2633960Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2634181Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19601
2022-11-23T01:51:49.2634397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19602
2022-11-23T01:51:49.2634783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2634959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2635344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2635518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2636013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2636190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2636571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2636759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2637069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2637313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2637714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2638275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2638522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2638753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2639013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnhzpqjey
2022-11-23T01:51:49.2639286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnhzpqjey/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2639541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6qc0ygej
2022-11-23T01:51:49.2639807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6qc0ygej/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2639910Z ok (6.267s)
2022-11-23T01:51:49.2639932Z 
2022-11-23T01:51:49.2640211Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2640325Z Ran 1 test in 6.267s
2022-11-23T01:51:49.2640346Z 
2022-11-23T01:51:49.2640419Z OK
2022-11-23T01:51:49.2640438Z 
2022-11-23T01:51:49.2640560Z Generating XML reports...
2022-11-23T01:51:49.2641015Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014323.xml
2022-11-23T01:51:49.2641388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2641564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2641951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2642142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2642161Z 
2022-11-23T01:51:49.2642269Z Running tests...
2022-11-23T01:51:49.2642582Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2642907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2643146Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2643363Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19716
2022-11-23T01:51:49.2643580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19717
2022-11-23T01:51:49.2643950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2644130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2644511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2644704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2645058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2645233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2645606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2645795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2646039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2646283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2646748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2647146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2647363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2647592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2647693Z ok (4.246s)
2022-11-23T01:51:49.2647713Z 
2022-11-23T01:51:49.2647979Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2648090Z Ran 1 test in 4.246s
2022-11-23T01:51:49.2648121Z 
2022-11-23T01:51:49.2648212Z OK
2022-11-23T01:51:49.2648232Z 
2022-11-23T01:51:49.2648355Z Generating XML reports...
2022-11-23T01:51:49.2648811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014331.xml
2022-11-23T01:51:49.2649185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2649343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2649727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2649917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2649937Z 
2022-11-23T01:51:49.2650043Z Running tests...
2022-11-23T01:51:49.2650307Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2650621Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2650860Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2651084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19825
2022-11-23T01:51:49.2651283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19826
2022-11-23T01:51:49.2651657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2651896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2652294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2652484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2652844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2653017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2653389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2653583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2653812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2654058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2654460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2654853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2655082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2655308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2655410Z ok (4.313s)
2022-11-23T01:51:49.2655482Z 
2022-11-23T01:51:49.2655754Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2655848Z Ran 1 test in 4.313s
2022-11-23T01:51:49.2655884Z 
2022-11-23T01:51:49.2655958Z OK
2022-11-23T01:51:49.2655977Z 
2022-11-23T01:51:49.2656100Z Generating XML reports...
2022-11-23T01:51:49.2656558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014338.xml
2022-11-23T01:51:49.2656928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2657101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2657479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2657667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2657687Z 
2022-11-23T01:51:49.2657801Z Running tests...
2022-11-23T01:51:49.2658046Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2658355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2658624Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2658846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19934
2022-11-23T01:51:49.2659064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19935
2022-11-23T01:51:49.2659436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2659610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2659989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2660168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2660534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2660706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2661130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2661325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2661572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2661816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2662217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2662615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2662832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2663170Z STAGE:2022-11-23 01:43:49 19935:19935 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2663400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2663736Z STAGE:2022-11-23 01:43:49 19934:19934 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2664071Z STAGE:2022-11-23 01:43:49 19934:19934 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2664418Z STAGE:2022-11-23 01:43:49 19934:19934 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2664751Z STAGE:2022-11-23 01:43:49 19935:19935 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2665095Z STAGE:2022-11-23 01:43:49 19935:19935 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2665257Z ok (4.408s)
2022-11-23T01:51:49.2665278Z 
2022-11-23T01:51:49.2665529Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2665643Z Ran 1 test in 4.408s
2022-11-23T01:51:49.2665662Z 
2022-11-23T01:51:49.2665754Z OK
2022-11-23T01:51:49.2665777Z 
2022-11-23T01:51:49.2665901Z Generating XML reports...
2022-11-23T01:51:49.2666356Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014345.xml
2022-11-23T01:51:49.2666727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2667110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2667507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2667704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2667724Z 
2022-11-23T01:51:49.2667815Z Running tests...
2022-11-23T01:51:49.2668079Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2668389Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2668653Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2668873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20047
2022-11-23T01:51:49.2669096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20048
2022-11-23T01:51:49.2669469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2669644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2670005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2670198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2670564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2670824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2671218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2671406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2671653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2671896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2672296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2672683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2672914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2673252Z STAGE:2022-11-23 01:43:56 20048:20048 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2673479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2673813Z STAGE:2022-11-23 01:43:56 20047:20047 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2674148Z STAGE:2022-11-23 01:43:56 20047:20047 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2674493Z STAGE:2022-11-23 01:43:56 20047:20047 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2674920Z STAGE:2022-11-23 01:43:56 20048:20048 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2675265Z STAGE:2022-11-23 01:43:56 20048:20048 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2675350Z ok (4.346s)
2022-11-23T01:51:49.2675370Z 
2022-11-23T01:51:49.2675638Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2675750Z Ran 1 test in 4.346s
2022-11-23T01:51:49.2675769Z 
2022-11-23T01:51:49.2675860Z OK
2022-11-23T01:51:49.2675880Z 
2022-11-23T01:51:49.2676004Z Generating XML reports...
2022-11-23T01:51:49.2676458Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014352.xml
2022-11-23T01:51:49.2676826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2677001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2677368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2677561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2677580Z 
2022-11-23T01:51:49.2677687Z Running tests...
2022-11-23T01:51:49.2677946Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2678262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2678546Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2678766Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20160
2022-11-23T01:51:49.2678981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20161
2022-11-23T01:51:49.2679351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2679513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2679890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2680079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2680494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2680679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2681058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2681250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2681496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2681724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2682132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2682528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2682761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2682991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2683230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2683468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2683862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2684317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2684541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:51:49.2684783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:51:49.2685174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.2685566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.2685798Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:51:49.2685902Z ok (22.177s)
2022-11-23T01:51:49.2685922Z 
2022-11-23T01:51:49.2686190Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2686307Z Ran 1 test in 22.177s
2022-11-23T01:51:49.2686327Z 
2022-11-23T01:51:49.2686419Z OK
2022-11-23T01:51:49.2686438Z 
2022-11-23T01:51:49.2686545Z Generating XML reports...
2022-11-23T01:51:49.2686998Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014359.xml
2022-11-23T01:51:49.2687370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2687546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2687972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2688170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2688191Z 
2022-11-23T01:51:49.2688300Z Running tests...
2022-11-23T01:51:49.2688565Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2688865Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2689193Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2689415Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20287
2022-11-23T01:51:49.2689688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20288
2022-11-23T01:51:49.2690068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2690243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2690618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2690808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2691175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2691336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2691711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2691901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2692144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2692387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2692786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2693182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2693471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2693700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2693922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2694163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2694560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2694952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2695192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:51:49.2695434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:51:49.2695824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.2696211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.2696446Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:51:49.2696659Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:51:49.2696763Z ok (22.271s)
2022-11-23T01:51:49.2696784Z 
2022-11-23T01:51:49.2697046Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2697158Z Ran 1 test in 22.271s
2022-11-23T01:51:49.2697177Z 
2022-11-23T01:51:49.2697269Z OK
2022-11-23T01:51:49.2697288Z 
2022-11-23T01:51:49.2697411Z Generating XML reports...
2022-11-23T01:51:49.2697861Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014424.xml
2022-11-23T01:51:49.2698242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2698418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2698836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2699036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2699056Z 
2022-11-23T01:51:49.2699164Z Running tests...
2022-11-23T01:51:49.2699431Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2699743Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2700023Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2700239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20414
2022-11-23T01:51:49.2700462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20415
2022-11-23T01:51:49.2700815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2700994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2701372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2701562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2701930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2702102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2702477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2702738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2702981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2703209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2703613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2704006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2704234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2704463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2704622Z skip: Skipped due to small world size. (4.339s)
2022-11-23T01:51:49.2704646Z 
2022-11-23T01:51:49.2704912Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2705024Z Ran 1 test in 4.339s
2022-11-23T01:51:49.2705043Z 
2022-11-23T01:51:49.2705132Z OK (skipped=1)
2022-11-23T01:51:49.2705169Z 
2022-11-23T01:51:49.2705274Z Generating XML reports...
2022-11-23T01:51:49.2705724Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014449.xml
2022-11-23T01:51:49.2706097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2706270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2706646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2707023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2707045Z 
2022-11-23T01:51:49.2707167Z Running tests...
2022-11-23T01:51:49.2707440Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2707734Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2708001Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2708304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20523
2022-11-23T01:51:49.2708535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20524
2022-11-23T01:51:49.2708910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2709086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2709465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2709660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2710031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2710188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2710567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2710755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2710997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2711241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2711640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2712033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2712339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2712551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2712787Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-11-23T01:51:49.2712888Z ok (6.455s)
2022-11-23T01:51:49.2712908Z 
2022-11-23T01:51:49.2713177Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2713288Z Ran 1 test in 6.455s
2022-11-23T01:51:49.2713308Z 
2022-11-23T01:51:49.2713400Z OK
2022-11-23T01:51:49.2713419Z 
2022-11-23T01:51:49.2713539Z Generating XML reports...
2022-11-23T01:51:49.2713991Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014455.xml
2022-11-23T01:51:49.2714361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2714526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2714907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2715099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2715119Z 
2022-11-23T01:51:49.2715224Z Running tests...
2022-11-23T01:51:49.2715491Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2715807Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2716090Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2716308Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20632
2022-11-23T01:51:49.2716512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20633
2022-11-23T01:51:49.2716884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2717057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2717502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2717702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2718077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2718250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2718625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2718817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2719049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2719295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2719699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2720095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2720325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2720552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2720793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2721032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2721490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2721862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2722082Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-11-23T01:51:49.2722255Z No ranks successfully processed in monitoredBarrier.
2022-11-23T01:51:49.2722479Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-11-23T01:51:49.2722581Z ok (4.354s)
2022-11-23T01:51:49.2722601Z 
2022-11-23T01:51:49.2722870Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2722981Z Ran 1 test in 4.354s
2022-11-23T01:51:49.2723001Z 
2022-11-23T01:51:49.2723091Z OK
2022-11-23T01:51:49.2723111Z 
2022-11-23T01:51:49.2723220Z Generating XML reports...
2022-11-23T01:51:49.2723675Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014504.xml
2022-11-23T01:51:49.2724048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2724226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2724602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2724791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2724811Z 
2022-11-23T01:51:49.2724919Z Running tests...
2022-11-23T01:51:49.2725183Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2725496Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2725760Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2725982Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20747
2022-11-23T01:51:49.2726199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20748
2022-11-23T01:51:49.2726622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2726802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2727177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2727366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2727732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2727888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2728272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2728461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2728705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2728950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2729351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2729745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2729972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2730196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2730472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2730715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2731119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2731515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2731743Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T01:51:49.2731845Z ok (4.461s)
2022-11-23T01:51:49.2731866Z 
2022-11-23T01:51:49.2732126Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2732236Z Ran 1 test in 4.461s
2022-11-23T01:51:49.2732256Z 
2022-11-23T01:51:49.2732346Z OK
2022-11-23T01:51:49.2732369Z 
2022-11-23T01:51:49.2732476Z Generating XML reports...
2022-11-23T01:51:49.2732928Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014511.xml
2022-11-23T01:51:49.2733300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2733481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2733862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2734058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2734078Z 
2022-11-23T01:51:49.2734185Z Running tests...
2022-11-23T01:51:49.2734447Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2734738Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2735023Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2735238Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20862
2022-11-23T01:51:49.2735453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20863
2022-11-23T01:51:49.2735874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2736058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2736437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2736626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2736991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2737148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2737528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2737716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2737964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2738208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2738605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2738999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2739228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2739454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2739655Z skip: Skipped due to small world size. (4.136s)
2022-11-23T01:51:49.2739675Z 
2022-11-23T01:51:49.2739944Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2740055Z Ran 1 test in 4.136s
2022-11-23T01:51:49.2740074Z 
2022-11-23T01:51:49.2740183Z OK (skipped=1)
2022-11-23T01:51:49.2740203Z 
2022-11-23T01:51:49.2740325Z Generating XML reports...
2022-11-23T01:51:49.2740774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014518.xml
2022-11-23T01:51:49.2741142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2741316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2741678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2741875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2741894Z 
2022-11-23T01:51:49.2742001Z Running tests...
2022-11-23T01:51:49.2742259Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2742573Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2742976Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:51:49.2742997Z 
2022-11-23T01:51:49.2743260Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2743370Z Ran 1 test in 0.003s
2022-11-23T01:51:49.2743390Z 
2022-11-23T01:51:49.2743497Z OK (skipped=1)
2022-11-23T01:51:49.2743517Z 
2022-11-23T01:51:49.2743622Z Generating XML reports...
2022-11-23T01:51:49.2744073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014525.xml
2022-11-23T01:51:49.2744450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2744624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2745059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2745256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2745277Z 
2022-11-23T01:51:49.2745384Z Running tests...
2022-11-23T01:51:49.2745646Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2745958Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2746341Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:51:49.2746366Z 
2022-11-23T01:51:49.2746628Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2746739Z Ran 1 test in 0.003s
2022-11-23T01:51:49.2746945Z 
2022-11-23T01:51:49.2747075Z OK (skipped=1)
2022-11-23T01:51:49.2747095Z 
2022-11-23T01:51:49.2747218Z Generating XML reports...
2022-11-23T01:51:49.2747681Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014527.xml
2022-11-23T01:51:49.2748052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2748225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2748602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2748776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2748796Z 
2022-11-23T01:51:49.2748902Z Running tests...
2022-11-23T01:51:49.2749269Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2749583Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2749984Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T01:51:49.2750009Z 
2022-11-23T01:51:49.2750266Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2750377Z Ran 1 test in 0.002s
2022-11-23T01:51:49.2750397Z 
2022-11-23T01:51:49.2750503Z OK (skipped=1)
2022-11-23T01:51:49.2750522Z 
2022-11-23T01:51:49.2750626Z Generating XML reports...
2022-11-23T01:51:49.2751075Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014530.xml
2022-11-23T01:51:49.2751446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2751629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2752013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2752203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2752223Z 
2022-11-23T01:51:49.2752333Z Running tests...
2022-11-23T01:51:49.2752591Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2752902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2753278Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-11-23T01:51:49.2753316Z 
2022-11-23T01:51:49.2753559Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2753668Z Ran 1 test in 0.003s
2022-11-23T01:51:49.2753687Z 
2022-11-23T01:51:49.2753797Z OK (skipped=1)
2022-11-23T01:51:49.2753816Z 
2022-11-23T01:51:49.2753937Z Generating XML reports...
2022-11-23T01:51:49.2754384Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014532.xml
2022-11-23T01:51:49.2754822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2755011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2755396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2755569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2755589Z 
2022-11-23T01:51:49.2755695Z Running tests...
2022-11-23T01:51:49.2755952Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2756264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2756566Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-11-23T01:51:49.2756586Z 
2022-11-23T01:51:49.2756846Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2756956Z Ran 1 test in 0.003s
2022-11-23T01:51:49.2756980Z 
2022-11-23T01:51:49.2757089Z OK (skipped=1)
2022-11-23T01:51:49.2757108Z 
2022-11-23T01:51:49.2757230Z Generating XML reports...
2022-11-23T01:51:49.2757661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014535.xml
2022-11-23T01:51:49.2758023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2758196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2758573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2758836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2758856Z 
2022-11-23T01:51:49.2758963Z Running tests...
2022-11-23T01:51:49.2759225Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2759544Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2759778Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:51:49.2759817Z 
2022-11-23T01:51:49.2760061Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2760172Z Ran 1 test in 0.002s
2022-11-23T01:51:49.2760192Z 
2022-11-23T01:51:49.2760298Z OK (skipped=1)
2022-11-23T01:51:49.2760317Z 
2022-11-23T01:51:49.2760439Z Generating XML reports...
2022-11-23T01:51:49.2760886Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014537.xml
2022-11-23T01:51:49.2761262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2761438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2761818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2761990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2762029Z 
2022-11-23T01:51:49.2762118Z Running tests...
2022-11-23T01:51:49.2762382Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2762693Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2762963Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:51:49.2762987Z 
2022-11-23T01:51:49.2763243Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2763354Z Ran 1 test in 0.003s
2022-11-23T01:51:49.2763374Z 
2022-11-23T01:51:49.2763478Z OK (skipped=1)
2022-11-23T01:51:49.2763497Z 
2022-11-23T01:51:49.2763617Z Generating XML reports...
2022-11-23T01:51:49.2764096Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014539.xml
2022-11-23T01:51:49.2764476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2764652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2765029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2765217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2765237Z 
2022-11-23T01:51:49.2765349Z Running tests...
2022-11-23T01:51:49.2765608Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2765917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2766213Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:51:49.2766250Z 
2022-11-23T01:51:49.2766495Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2766601Z Ran 1 test in 0.002s
2022-11-23T01:51:49.2766620Z 
2022-11-23T01:51:49.2766726Z OK (skipped=1)
2022-11-23T01:51:49.2766745Z 
2022-11-23T01:51:49.2766863Z Generating XML reports...
2022-11-23T01:51:49.2767310Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014542.xml
2022-11-23T01:51:49.2767676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2767962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2768346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2768518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2768556Z 
2022-11-23T01:51:49.2768650Z Running tests...
2022-11-23T01:51:49.2768911Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2769218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2769516Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2769735Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21235
2022-11-23T01:51:49.2769947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21236
2022-11-23T01:51:49.2770324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2770500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2770862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2771050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2771416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2771588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2771962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2772151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2772400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2772644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2773045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2773477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2773717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2773943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2774044Z ok (4.374s)
2022-11-23T01:51:49.2774064Z 
2022-11-23T01:51:49.2774332Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2774442Z Ran 1 test in 4.374s
2022-11-23T01:51:49.2774466Z 
2022-11-23T01:51:49.2774555Z OK
2022-11-23T01:51:49.2774574Z 
2022-11-23T01:51:49.2774695Z Generating XML reports...
2022-11-23T01:51:49.2775131Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014544.xml
2022-11-23T01:51:49.2775505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2775680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2776059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2776248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2776268Z 
2022-11-23T01:51:49.2776374Z Running tests...
2022-11-23T01:51:49.2776635Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2776947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2777301Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2777502Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21344
2022-11-23T01:51:49.2777724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21345
2022-11-23T01:51:49.2778098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2778271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2778647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2778838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2779203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2779382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2779739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2779928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2780172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2780415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2780814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2781205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2781432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2781661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2781760Z ok (4.255s)
2022-11-23T01:51:49.2781781Z 
2022-11-23T01:51:49.2782027Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2782137Z Ran 1 test in 4.255s
2022-11-23T01:51:49.2782156Z 
2022-11-23T01:51:49.2782311Z OK
2022-11-23T01:51:49.2782333Z 
2022-11-23T01:51:49.2782460Z Generating XML reports...
2022-11-23T01:51:49.2782916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014551.xml
2022-11-23T01:51:49.2783283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2783456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2783831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2784026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2784046Z 
2022-11-23T01:51:49.2784137Z Running tests...
2022-11-23T01:51:49.2784395Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2784710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2784987Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:51:49.2785006Z 
2022-11-23T01:51:49.2785261Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2785370Z Ran 1 test in 0.002s
2022-11-23T01:51:49.2785390Z 
2022-11-23T01:51:49.2785494Z OK (skipped=1)
2022-11-23T01:51:49.2785514Z 
2022-11-23T01:51:49.2785633Z Generating XML reports...
2022-11-23T01:51:49.2786065Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014558.xml
2022-11-23T01:51:49.2786498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2786673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2787256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2787449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2787469Z 
2022-11-23T01:51:49.2787576Z Running tests...
2022-11-23T01:51:49.2787843Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2788154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2788454Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T01:51:49.2788478Z 
2022-11-23T01:51:49.2788723Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2788832Z Ran 1 test in 0.002s
2022-11-23T01:51:49.2788852Z 
2022-11-23T01:51:49.2788985Z OK (skipped=1)
2022-11-23T01:51:49.2789005Z 
2022-11-23T01:51:49.2789121Z Generating XML reports...
2022-11-23T01:51:49.2789573Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014600.xml
2022-11-23T01:51:49.2789942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2790113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2790487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2790675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2790694Z 
2022-11-23T01:51:49.2790788Z Running tests...
2022-11-23T01:51:49.2791046Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2791357Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2791632Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2792468Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.629s)
2022-11-23T01:51:49.2792492Z 
2022-11-23T01:51:49.2792761Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2792873Z Ran 1 test in 1.629s
2022-11-23T01:51:49.2792892Z 
2022-11-23T01:51:49.2792999Z OK (skipped=1)
2022-11-23T01:51:49.2793018Z 
2022-11-23T01:51:49.2793143Z Generating XML reports...
2022-11-23T01:51:49.2793594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014603.xml
2022-11-23T01:51:49.2793945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2794124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2794502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2794688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2794708Z 
2022-11-23T01:51:49.2794812Z Running tests...
2022-11-23T01:51:49.2795071Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2795379Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2795746Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2795950Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21553
2022-11-23T01:51:49.2796167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21554
2022-11-23T01:51:49.2796543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2796716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2797090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2797274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2797639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2797811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2798191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2798363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2798611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2798857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2799253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2799646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2799875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2800102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2800362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeahfvujf
2022-11-23T01:51:49.2800616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeahfvujf/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2800922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyybpm4yl
2022-11-23T01:51:49.2801198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyybpm4yl/_remote_module_non_scriptable.py
2022-11-23T01:51:49.2801296Z ok (6.244s)
2022-11-23T01:51:49.2801316Z 
2022-11-23T01:51:49.2801590Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2801699Z Ran 1 test in 6.244s
2022-11-23T01:51:49.2801719Z 
2022-11-23T01:51:49.2801809Z OK
2022-11-23T01:51:49.2801828Z 
2022-11-23T01:51:49.2801949Z Generating XML reports...
2022-11-23T01:51:49.2802400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014607.xml
2022-11-23T01:51:49.2802759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2802932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2803310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2803499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2803520Z 
2022-11-23T01:51:49.2803624Z Running tests...
2022-11-23T01:51:49.2803885Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2804197Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2804466Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2804728Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21668
2022-11-23T01:51:49.2804946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21669
2022-11-23T01:51:49.2805318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2805493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2805866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2806054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2806416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2806591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2806966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2807141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2807386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2807632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2808034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2808432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2808657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2808880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2808978Z ok (6.828s)
2022-11-23T01:51:49.2808999Z 
2022-11-23T01:51:49.2809264Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2809358Z Ran 1 test in 6.828s
2022-11-23T01:51:49.2809377Z 
2022-11-23T01:51:49.2809467Z OK
2022-11-23T01:51:49.2809486Z 
2022-11-23T01:51:49.2809606Z Generating XML reports...
2022-11-23T01:51:49.2810103Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014615.xml
2022-11-23T01:51:49.2810488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2810665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2811038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2811224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2811244Z 
2022-11-23T01:51:49.2811334Z Running tests...
2022-11-23T01:51:49.2811598Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2811905Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2812188Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2812404Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21780
2022-11-23T01:51:49.2812621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21781
2022-11-23T01:51:49.2812992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2813162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2813536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2813709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2814137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2814307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2814685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2814873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2815118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2815361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2815757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2816133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2816367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2816594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2816693Z ok (6.955s)
2022-11-23T01:51:49.2816713Z 
2022-11-23T01:51:49.2816979Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2817089Z Ran 1 test in 6.955s
2022-11-23T01:51:49.2817109Z 
2022-11-23T01:51:49.2817201Z OK
2022-11-23T01:51:49.2817220Z 
2022-11-23T01:51:49.2817343Z Generating XML reports...
2022-11-23T01:51:49.2817794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014625.xml
2022-11-23T01:51:49.2818147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2818323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2818707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2818895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2818915Z 
2022-11-23T01:51:49.2819018Z Running tests...
2022-11-23T01:51:49.2819330Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2819653Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2819931Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2820671Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.597s)
2022-11-23T01:51:49.2820697Z 
2022-11-23T01:51:49.2820957Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2821051Z Ran 1 test in 1.598s
2022-11-23T01:51:49.2821071Z 
2022-11-23T01:51:49.2821174Z OK (skipped=1)
2022-11-23T01:51:49.2821193Z 
2022-11-23T01:51:49.2821313Z Generating XML reports...
2022-11-23T01:51:49.2821765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014634.xml
2022-11-23T01:51:49.2822127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2822301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2822676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2822864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2822939Z 
2022-11-23T01:51:49.2823032Z Running tests...
2022-11-23T01:51:49.2823298Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2823603Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2823902Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2824638Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.625s)
2022-11-23T01:51:49.2824660Z 
2022-11-23T01:51:49.2824919Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2825029Z Ran 1 test in 1.625s
2022-11-23T01:51:49.2825048Z 
2022-11-23T01:51:49.2825152Z OK (skipped=1)
2022-11-23T01:51:49.2825171Z 
2022-11-23T01:51:49.2825293Z Generating XML reports...
2022-11-23T01:51:49.2825745Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014638.xml
2022-11-23T01:51:49.2826102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2826279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2826660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2827034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2827056Z 
2022-11-23T01:51:49.2827171Z Running tests...
2022-11-23T01:51:49.2827438Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2827749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2828063Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2828282Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21960
2022-11-23T01:51:49.2828566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21961
2022-11-23T01:51:49.2828956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2829130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2829504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2829688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2830047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2830228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2830710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2830884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2831133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2831373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2831772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2832169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2832394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2832695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2832843Z skip: Need at least 4 CUDA devices (4.235s)
2022-11-23T01:51:49.2832864Z 
2022-11-23T01:51:49.2833128Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2833222Z Ran 1 test in 4.235s
2022-11-23T01:51:49.2833246Z 
2022-11-23T01:51:49.2833350Z OK (skipped=1)
2022-11-23T01:51:49.2833370Z 
2022-11-23T01:51:49.2833490Z Generating XML reports...
2022-11-23T01:51:49.2833942Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014643.xml
2022-11-23T01:51:49.2834323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2834633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2835019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2835213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2835233Z 
2022-11-23T01:51:49.2835336Z Running tests...
2022-11-23T01:51:49.2835584Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2835899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2836224Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2836439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22069
2022-11-23T01:51:49.2836655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22070
2022-11-23T01:51:49.2837022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2837199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2837575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2837750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2838179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2838362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2838739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2838927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2839168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2839408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2839810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2840205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2840419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2840644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2840791Z skip: Need at least 4 CUDA devices (4.262s)
2022-11-23T01:51:49.2840811Z 
2022-11-23T01:51:49.2841071Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2841181Z Ran 1 test in 4.262s
2022-11-23T01:51:49.2841200Z 
2022-11-23T01:51:49.2841304Z OK (skipped=1)
2022-11-23T01:51:49.2841324Z 
2022-11-23T01:51:49.2841440Z Generating XML reports...
2022-11-23T01:51:49.2841960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014649.xml
2022-11-23T01:51:49.2842326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2842485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2842864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2843047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2843067Z 
2022-11-23T01:51:49.2843171Z Running tests...
2022-11-23T01:51:49.2843426Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2843736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2844013Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2844766Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.615s)
2022-11-23T01:51:49.2844791Z 
2022-11-23T01:51:49.2845051Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2845145Z Ran 1 test in 1.615s
2022-11-23T01:51:49.2845176Z 
2022-11-23T01:51:49.2845266Z OK (skipped=1)
2022-11-23T01:51:49.2845286Z 
2022-11-23T01:51:49.2845407Z Generating XML reports...
2022-11-23T01:51:49.2845848Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014656.xml
2022-11-23T01:51:49.2846216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2846394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2846773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2846960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2846980Z 
2022-11-23T01:51:49.2847150Z Running tests...
2022-11-23T01:51:49.2847406Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2847717Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2847977Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2848195Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22212
2022-11-23T01:51:49.2848405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22213
2022-11-23T01:51:49.2848778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2848951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2849327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2849503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2849868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2850039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2850412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2850602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2850844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2851149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2851547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2851947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2852161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2852393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2852631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2852872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2853264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2853657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2853993Z STAGE:2022-11-23 01:47:04 22212:22212 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2854317Z STAGE:2022-11-23 01:47:04 22213:22213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2854647Z STAGE:2022-11-23 01:47:04 22213:22213 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2854955Z STAGE:2022-11-23 01:47:04 22212:22212 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2855299Z STAGE:2022-11-23 01:47:04 22213:22213 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2855639Z STAGE:2022-11-23 01:47:04 22212:22212 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2855962Z STAGE:2022-11-23 01:47:04 22213:22213 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2856276Z STAGE:2022-11-23 01:47:04 22212:22212 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2856651Z STAGE:2022-11-23 01:47:04 22212:22212 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2856988Z STAGE:2022-11-23 01:47:04 22213:22213 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2857328Z STAGE:2022-11-23 01:47:04 22212:22212 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2857663Z STAGE:2022-11-23 01:47:04 22213:22213 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2857748Z ok (4.253s)
2022-11-23T01:51:49.2857768Z 
2022-11-23T01:51:49.2858029Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2858142Z Ran 1 test in 4.254s
2022-11-23T01:51:49.2858162Z 
2022-11-23T01:51:49.2858252Z OK
2022-11-23T01:51:49.2858271Z 
2022-11-23T01:51:49.2858394Z Generating XML reports...
2022-11-23T01:51:49.2858845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014700.xml
2022-11-23T01:51:49.2859214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2859389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2859767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2859943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2859962Z 
2022-11-23T01:51:49.2860063Z Running tests...
2022-11-23T01:51:49.2860325Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2860697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2860958Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2861176Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22331
2022-11-23T01:51:49.2861395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22332
2022-11-23T01:51:49.2861764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2861922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2862295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2862482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2862844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2863018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2863397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2863587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2863838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2864080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2864466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2864857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2865085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2865314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2865551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2865838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2866246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2866632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2867161Z STAGE:2022-11-23 01:47:11 22331:22331 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2867481Z STAGE:2022-11-23 01:47:11 22332:22332 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2868030Z STAGE:2022-11-23 01:47:11 22331:22331 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:47:11 22332:22332 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2868051Z 
2022-11-23T01:51:49.2868399Z STAGE:2022-11-23 01:47:11 22331:22331 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2868745Z STAGE:2022-11-23 01:47:11 22332:22332 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2869068Z STAGE:2022-11-23 01:47:11 22332:22332 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2869383Z STAGE:2022-11-23 01:47:11 22331:22331 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2869713Z STAGE:2022-11-23 01:47:11 22332:22332 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2870034Z STAGE:2022-11-23 01:47:11 22331:22331 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2870478Z STAGE:2022-11-23 01:47:11 22332:22332 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2870802Z STAGE:2022-11-23 01:47:11 22331:22331 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2870902Z ok (4.360s)
2022-11-23T01:51:49.2870922Z 
2022-11-23T01:51:49.2871185Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2871295Z Ran 1 test in 4.360s
2022-11-23T01:51:49.2871315Z 
2022-11-23T01:51:49.2871400Z OK
2022-11-23T01:51:49.2871420Z 
2022-11-23T01:51:49.2871539Z Generating XML reports...
2022-11-23T01:51:49.2871990Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014707.xml
2022-11-23T01:51:49.2872355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2872529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2872898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2873087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2873107Z 
2022-11-23T01:51:49.2873211Z Running tests...
2022-11-23T01:51:49.2873472Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2873787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2874054Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2874269Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22450
2022-11-23T01:51:49.2874483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22451
2022-11-23T01:51:49.2874835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2875015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2875393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2875583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2876013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2876197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2876581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2876770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2877017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2877246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2877648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2878043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2878273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2878501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2878741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2878977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2879372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2879825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2880143Z STAGE:2022-11-23 01:47:18 22450:22450 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2880469Z STAGE:2022-11-23 01:47:18 22451:22451 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2881016Z STAGE:2022-11-23 01:47:18 22450:22450 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:47:18 22451:22451 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2881036Z 
2022-11-23T01:51:49.2881605Z STAGE:2022-11-23 01:47:18 22450:22450 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:47:18 22451:22451 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2881626Z 
2022-11-23T01:51:49.2881948Z STAGE:2022-11-23 01:47:18 22451:22451 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2882274Z STAGE:2022-11-23 01:47:18 22450:22450 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2882602Z STAGE:2022-11-23 01:47:18 22451:22451 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2882928Z STAGE:2022-11-23 01:47:18 22450:22450 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2883269Z STAGE:2022-11-23 01:47:18 22451:22451 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2883606Z STAGE:2022-11-23 01:47:18 22450:22450 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2883706Z ok (4.244s)
2022-11-23T01:51:49.2883726Z 
2022-11-23T01:51:49.2883971Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2884079Z Ran 1 test in 4.244s
2022-11-23T01:51:49.2884098Z 
2022-11-23T01:51:49.2884191Z OK
2022-11-23T01:51:49.2884210Z 
2022-11-23T01:51:49.2884332Z Generating XML reports...
2022-11-23T01:51:49.2884781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014714.xml
2022-11-23T01:51:49.2885152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2885381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2885769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2885943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2885978Z 
2022-11-23T01:51:49.2886068Z Running tests...
2022-11-23T01:51:49.2886329Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2886640Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2886908Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2887126Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22569
2022-11-23T01:51:49.2887337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22570
2022-11-23T01:51:49.2887710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2887885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2888250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2888439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2888801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2889076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2889460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2889646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2889891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2890130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2890510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2890906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2891134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2891360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2891601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.2891842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.2892242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2892630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.2892965Z STAGE:2022-11-23 01:47:25 22570:22570 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2893270Z STAGE:2022-11-23 01:47:25 22569:22569 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2893811Z STAGE:2022-11-23 01:47:25 22570:22570 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:47:25 22569:22569 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2893837Z 
2022-11-23T01:51:49.2894400Z STAGE:2022-11-23 01:47:25 22570:22570 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:47:25 22569:22569 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2894475Z 
2022-11-23T01:51:49.2894815Z STAGE:2022-11-23 01:47:25 22570:22570 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2895133Z STAGE:2022-11-23 01:47:25 22569:22569 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2895463Z STAGE:2022-11-23 01:47:25 22569:22569 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2895787Z STAGE:2022-11-23 01:47:25 22570:22570 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2896129Z STAGE:2022-11-23 01:47:25 22569:22569 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2896478Z STAGE:2022-11-23 01:47:25 22570:22570 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2896579Z ok (4.386s)
2022-11-23T01:51:49.2896600Z 
2022-11-23T01:51:49.2896860Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2896958Z Ran 1 test in 4.387s
2022-11-23T01:51:49.2896978Z 
2022-11-23T01:51:49.2897070Z OK
2022-11-23T01:51:49.2897089Z 
2022-11-23T01:51:49.2897209Z Generating XML reports...
2022-11-23T01:51:49.2897658Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014721.xml
2022-11-23T01:51:49.2898024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2898195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2898575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2898823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2898843Z 
2022-11-23T01:51:49.2898934Z Running tests...
2022-11-23T01:51:49.2899196Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2899510Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2899768Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2899982Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22688
2022-11-23T01:51:49.2900195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22689
2022-11-23T01:51:49.2900563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2900736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2901115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2901287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2901650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2901822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2902194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2902381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2902621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2902864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2903264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2903644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2903924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2904158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2904313Z skip: Skipped due to small world size. (4.244s)
2022-11-23T01:51:49.2904334Z 
2022-11-23T01:51:49.2904599Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2904709Z Ran 1 test in 4.244s
2022-11-23T01:51:49.2904729Z 
2022-11-23T01:51:49.2904835Z OK (skipped=1)
2022-11-23T01:51:49.2904854Z 
2022-11-23T01:51:49.2904976Z Generating XML reports...
2022-11-23T01:51:49.2905422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014728.xml
2022-11-23T01:51:49.2905777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2905950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2906331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2906522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2906542Z 
2022-11-23T01:51:49.2906650Z Running tests...
2022-11-23T01:51:49.2907167Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2907497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2907754Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2908067Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22797
2022-11-23T01:51:49.2908268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22798
2022-11-23T01:51:49.2908637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2908811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2909182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2909368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2909730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2909902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2910271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2910446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2910688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2910931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2911330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2911720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2911948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2912174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2912330Z skip: Skipped due to small world size. (4.249s)
2022-11-23T01:51:49.2912355Z 
2022-11-23T01:51:49.2912617Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2912710Z Ran 1 test in 4.249s
2022-11-23T01:51:49.2912729Z 
2022-11-23T01:51:49.2912831Z OK (skipped=1)
2022-11-23T01:51:49.2912850Z 
2022-11-23T01:51:49.2912971Z Generating XML reports...
2022-11-23T01:51:49.2913561Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014735.xml
2022-11-23T01:51:49.2913952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2914127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2914498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2914685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2914710Z 
2022-11-23T01:51:49.2914817Z Running tests...
2022-11-23T01:51:49.2915063Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2915373Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2915640Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2915858Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22906
2022-11-23T01:51:49.2916073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22907
2022-11-23T01:51:49.2916444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2916617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2916991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2917225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2917597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2917769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2918149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2918335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2918578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2918821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2919218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2919621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2919832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2920057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2920216Z skip: Skipped due to small world size. (4.353s)
2022-11-23T01:51:49.2920237Z 
2022-11-23T01:51:49.2920497Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2920607Z Ran 1 test in 4.353s
2022-11-23T01:51:49.2920627Z 
2022-11-23T01:51:49.2920733Z OK (skipped=1)
2022-11-23T01:51:49.2920752Z 
2022-11-23T01:51:49.2920873Z Generating XML reports...
2022-11-23T01:51:49.2921326Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014741.xml
2022-11-23T01:51:49.2921679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2921860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2922236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2922424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2922494Z 
2022-11-23T01:51:49.2922605Z Running tests...
2022-11-23T01:51:49.2922868Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2923179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2923433Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2923649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23015
2022-11-23T01:51:49.2923848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23016
2022-11-23T01:51:49.2924220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2924392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2924772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2924960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2925328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2925500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2925871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2926044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2926346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2926590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2926991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2927388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2927614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2927838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2927990Z skip: Skipped due to small world size. (4.132s)
2022-11-23T01:51:49.2928009Z 
2022-11-23T01:51:49.2928278Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2928371Z Ran 1 test in 4.133s
2022-11-23T01:51:49.2928411Z 
2022-11-23T01:51:49.2928501Z OK (skipped=1)
2022-11-23T01:51:49.2928519Z 
2022-11-23T01:51:49.2928639Z Generating XML reports...
2022-11-23T01:51:49.2929081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014748.xml
2022-11-23T01:51:49.2929453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2929624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2929999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2930185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2930205Z 
2022-11-23T01:51:49.2930310Z Running tests...
2022-11-23T01:51:49.2930553Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2930861Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2931110Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2931326Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23124
2022-11-23T01:51:49.2931593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23125
2022-11-23T01:51:49.2931977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2932149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2932525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2932699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2933066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2933242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2933618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2933803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2934047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2934290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2934692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2935083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2935295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2935579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2935919Z STAGE:2022-11-23 01:47:59 23124:23124 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2936238Z STAGE:2022-11-23 01:47:59 23125:23125 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2936775Z STAGE:2022-11-23 01:47:59 23124:23124 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:47:59 23125:23125 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2936796Z 
2022-11-23T01:51:49.2937358Z STAGE:2022-11-23 01:47:59 23124:23124 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:47:59 23125:23125 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2937378Z 
2022-11-23T01:51:49.2937700Z STAGE:2022-11-23 01:47:59 23125:23125 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2938025Z STAGE:2022-11-23 01:47:59 23124:23124 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2938356Z STAGE:2022-11-23 01:47:59 23124:23124 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2938686Z STAGE:2022-11-23 01:47:59 23125:23125 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2939028Z STAGE:2022-11-23 01:47:59 23124:23124 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2939355Z STAGE:2022-11-23 01:47:59 23125:23125 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2939452Z ok (4.239s)
2022-11-23T01:51:49.2939472Z 
2022-11-23T01:51:49.2939731Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2939836Z Ran 1 test in 4.240s
2022-11-23T01:51:49.2939856Z 
2022-11-23T01:51:49.2939946Z OK
2022-11-23T01:51:49.2939969Z 
2022-11-23T01:51:49.2940090Z Generating XML reports...
2022-11-23T01:51:49.2940539Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014755.xml
2022-11-23T01:51:49.2940910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2941120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2941510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2941699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2941719Z 
2022-11-23T01:51:49.2941824Z Running tests...
2022-11-23T01:51:49.2942085Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2942395Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2942646Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2942865Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23237
2022-11-23T01:51:49.2943063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23238
2022-11-23T01:51:49.2943442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2943615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2943979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2944151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2944524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2944712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2945155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2945342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2945568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2945817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2946216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2946606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2947076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2947428Z STAGE:2022-11-23 01:48:06 23238:23238 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2947662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2947994Z STAGE:2022-11-23 01:48:06 23237:23237 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2948329Z STAGE:2022-11-23 01:48:06 23238:23238 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2948638Z STAGE:2022-11-23 01:48:06 23237:23237 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2948979Z STAGE:2022-11-23 01:48:06 23238:23238 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2949319Z STAGE:2022-11-23 01:48:06 23237:23237 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2949639Z STAGE:2022-11-23 01:48:06 23238:23238 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2949952Z STAGE:2022-11-23 01:48:06 23237:23237 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2950491Z STAGE:2022-11-23 01:48:06 23237:23237 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:48:06 23238:23238 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2950513Z 
2022-11-23T01:51:49.2950933Z STAGE:2022-11-23 01:48:06 23238:23238 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2951297Z STAGE:2022-11-23 01:48:06 23237:23237 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2951395Z ok (4.352s)
2022-11-23T01:51:49.2951415Z 
2022-11-23T01:51:49.2951676Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2951769Z Ran 1 test in 4.352s
2022-11-23T01:51:49.2951789Z 
2022-11-23T01:51:49.2951877Z OK
2022-11-23T01:51:49.2951896Z 
2022-11-23T01:51:49.2952015Z Generating XML reports...
2022-11-23T01:51:49.2952469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014802.xml
2022-11-23T01:51:49.2952845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2953019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2953404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2953592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2953612Z 
2022-11-23T01:51:49.2953702Z Running tests...
2022-11-23T01:51:49.2953960Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2954274Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2954550Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-11-23T01:51:49.2954637Z 
2022-11-23T01:51:49.2954912Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2955023Z Ran 1 test in 0.002s
2022-11-23T01:51:49.2955043Z 
2022-11-23T01:51:49.2955149Z OK (skipped=1)
2022-11-23T01:51:49.2955168Z 
2022-11-23T01:51:49.2955289Z Generating XML reports...
2022-11-23T01:51:49.2955748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014809.xml
2022-11-23T01:51:49.2956104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2956277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2956650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2956834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2956858Z 
2022-11-23T01:51:49.2956961Z Running tests...
2022-11-23T01:51:49.2957219Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2957524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2957778Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2957983Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23383
2022-11-23T01:51:49.2958202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23384
2022-11-23T01:51:49.2958571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2958745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2959124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2959317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2959683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2959855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2960284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2960466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2960708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2960948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2961352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2961747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2961982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2962314Z STAGE:2022-11-23 01:48:15 23384:23384 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2962545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2962877Z STAGE:2022-11-23 01:48:15 23383:23383 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2963194Z STAGE:2022-11-23 01:48:15 23384:23384 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2963520Z STAGE:2022-11-23 01:48:15 23383:23383 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2964085Z STAGE:2022-11-23 01:48:15 23384:23384 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:48:15 23383:23383 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2964156Z 
2022-11-23T01:51:49.2964488Z STAGE:2022-11-23 01:48:15 23384:23384 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2964805Z STAGE:2022-11-23 01:48:15 23383:23383 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2965129Z STAGE:2022-11-23 01:48:15 23384:23384 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2965453Z STAGE:2022-11-23 01:48:15 23383:23383 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2965791Z STAGE:2022-11-23 01:48:15 23384:23384 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2966132Z STAGE:2022-11-23 01:48:15 23383:23383 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2966215Z ok (4.360s)
2022-11-23T01:51:49.2966252Z 
2022-11-23T01:51:49.2966499Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2966612Z Ran 1 test in 4.361s
2022-11-23T01:51:49.2966632Z 
2022-11-23T01:51:49.2966721Z OK
2022-11-23T01:51:49.2966740Z 
2022-11-23T01:51:49.2966862Z Generating XML reports...
2022-11-23T01:51:49.2967320Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014811.xml
2022-11-23T01:51:49.2967691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2967865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2968243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2968417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2968453Z 
2022-11-23T01:51:49.2968543Z Running tests...
2022-11-23T01:51:49.2968804Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2969122Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2969415Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2022-11-23T01:51:49.2969435Z 
2022-11-23T01:51:49.2969746Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2969859Z Ran 1 test in 0.003s
2022-11-23T01:51:49.2969879Z 
2022-11-23T01:51:49.2969985Z OK (skipped=1)
2022-11-23T01:51:49.2970004Z 
2022-11-23T01:51:49.2970124Z Generating XML reports...
2022-11-23T01:51:49.2970557Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014818.xml
2022-11-23T01:51:49.2970930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2971102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2971486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2971675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2971694Z 
2022-11-23T01:51:49.2971800Z Running tests...
2022-11-23T01:51:49.2972065Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2972376Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2972631Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-11-23T01:51:49.2972669Z 
2022-11-23T01:51:49.2972912Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2973022Z Ran 1 test in 0.003s
2022-11-23T01:51:49.2973042Z 
2022-11-23T01:51:49.2973145Z OK (skipped=1)
2022-11-23T01:51:49.2973222Z 
2022-11-23T01:51:49.2973350Z Generating XML reports...
2022-11-23T01:51:49.2973801Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014820.xml
2022-11-23T01:51:49.2974166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2974344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2974723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2974895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2974930Z 
2022-11-23T01:51:49.2975019Z Running tests...
2022-11-23T01:51:49.2975282Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2975596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2975845Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2976062Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23562
2022-11-23T01:51:49.2976276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23563
2022-11-23T01:51:49.2976646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2976821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2977182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2977373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2977738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2977910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2978285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2978474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2978715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2979022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2979418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2979814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2980037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2980373Z STAGE:2022-11-23 01:48:27 23563:23563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2980607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2980937Z STAGE:2022-11-23 01:48:27 23562:23562 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2981479Z STAGE:2022-11-23 01:48:27 23562:23562 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:48:27 23563:23563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2981500Z 
2022-11-23T01:51:49.2982064Z STAGE:2022-11-23 01:48:27 23563:23563 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:48:27 23562:23562 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2982084Z 
2022-11-23T01:51:49.2982407Z STAGE:2022-11-23 01:48:27 23563:23563 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2982725Z STAGE:2022-11-23 01:48:27 23562:23562 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2983126Z STAGE:2022-11-23 01:48:27 23563:23563 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2983440Z STAGE:2022-11-23 01:48:27 23562:23562 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2983789Z STAGE:2022-11-23 01:48:27 23563:23563 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2984135Z STAGE:2022-11-23 01:48:27 23562:23562 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.2984236Z ok (4.281s)
2022-11-23T01:51:49.2984256Z 
2022-11-23T01:51:49.2984521Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2984630Z Ran 1 test in 4.281s
2022-11-23T01:51:49.2984650Z 
2022-11-23T01:51:49.2984739Z OK
2022-11-23T01:51:49.2984758Z 
2022-11-23T01:51:49.2984883Z Generating XML reports...
2022-11-23T01:51:49.2985317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014823.xml
2022-11-23T01:51:49.2985691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2985866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2986248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2986433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2986453Z 
2022-11-23T01:51:49.2986558Z Running tests...
2022-11-23T01:51:49.2987051Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2987384Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2987638Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-11-23T01:51:49.2987664Z 
2022-11-23T01:51:49.2987904Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2988059Z Ran 1 test in 0.002s
2022-11-23T01:51:49.2988079Z 
2022-11-23T01:51:49.2988186Z OK (skipped=1)
2022-11-23T01:51:49.2988205Z 
2022-11-23T01:51:49.2988325Z Generating XML reports...
2022-11-23T01:51:49.2988863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014830.xml
2022-11-23T01:51:49.2989285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2989463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2989839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2990026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2990045Z 
2022-11-23T01:51:49.2990139Z Running tests...
2022-11-23T01:51:49.2990401Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2990710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2990972Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-11-23T01:51:49.2990996Z 
2022-11-23T01:51:49.2991253Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2991361Z Ran 1 test in 0.002s
2022-11-23T01:51:49.2991380Z 
2022-11-23T01:51:49.2991481Z OK (skipped=1)
2022-11-23T01:51:49.2991500Z 
2022-11-23T01:51:49.2991619Z Generating XML reports...
2022-11-23T01:51:49.2992052Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014832.xml
2022-11-23T01:51:49.2992426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2992678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2993059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2993248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2993267Z 
2022-11-23T01:51:49.2993377Z Running tests...
2022-11-23T01:51:49.2993635Z ----------------------------------------------------------------------
2022-11-23T01:51:49.2993946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.2994202Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.2994405Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23741
2022-11-23T01:51:49.2994620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23742
2022-11-23T01:51:49.2994993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2995165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2995541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2995731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2996096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.2996271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.2996648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.2996820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.2997067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.2997315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.2997717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2998167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.2998403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.2998737Z STAGE:2022-11-23 01:48:38 23742:23742 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2998963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.2999276Z STAGE:2022-11-23 01:48:38 23741:23741 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.2999616Z STAGE:2022-11-23 01:48:38 23742:23742 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.2999938Z STAGE:2022-11-23 01:48:38 23741:23741 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3000280Z STAGE:2022-11-23 01:48:38 23742:23742 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3000627Z STAGE:2022-11-23 01:48:38 23741:23741 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3000948Z STAGE:2022-11-23 01:48:38 23742:23742 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3001264Z STAGE:2022-11-23 01:48:38 23741:23741 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3001589Z STAGE:2022-11-23 01:48:38 23741:23741 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3001911Z STAGE:2022-11-23 01:48:38 23742:23742 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3002297Z STAGE:2022-11-23 01:48:38 23741:23741 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3002634Z STAGE:2022-11-23 01:48:38 23742:23742 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3002736Z ok (4.264s)
2022-11-23T01:51:49.3002756Z 
2022-11-23T01:51:49.3003016Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3003124Z Ran 1 test in 4.264s
2022-11-23T01:51:49.3003144Z 
2022-11-23T01:51:49.3003232Z OK
2022-11-23T01:51:49.3003252Z 
2022-11-23T01:51:49.3003372Z Generating XML reports...
2022-11-23T01:51:49.3003822Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014834.xml
2022-11-23T01:51:49.3004192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3004350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3004735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3004924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3004943Z 
2022-11-23T01:51:49.3005049Z Running tests...
2022-11-23T01:51:49.3005314Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3005626Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3005864Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3006081Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23854
2022-11-23T01:51:49.3006282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23855
2022-11-23T01:51:49.3006653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3006832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3007208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3007393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3007806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3007988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3008371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3008557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3008787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3009030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3009434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3009821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3010057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3010383Z STAGE:2022-11-23 01:48:45 23855:23855 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3010611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3010939Z STAGE:2022-11-23 01:48:45 23854:23854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3011273Z STAGE:2022-11-23 01:48:45 23854:23854 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3011648Z STAGE:2022-11-23 01:48:45 23855:23855 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3011991Z STAGE:2022-11-23 01:48:45 23854:23854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3012334Z STAGE:2022-11-23 01:48:45 23855:23855 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3012863Z STAGE:2022-11-23 01:48:45 23855:23855 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:48:45 23854:23854 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3012884Z 
2022-11-23T01:51:49.3013211Z STAGE:2022-11-23 01:48:45 23855:23855 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3013534Z STAGE:2022-11-23 01:48:45 23854:23854 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3013869Z STAGE:2022-11-23 01:48:45 23855:23855 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3014212Z STAGE:2022-11-23 01:48:45 23854:23854 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3014312Z ok (4.244s)
2022-11-23T01:51:49.3014332Z 
2022-11-23T01:51:49.3014576Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3014688Z Ran 1 test in 4.244s
2022-11-23T01:51:49.3014711Z 
2022-11-23T01:51:49.3014801Z OK
2022-11-23T01:51:49.3014821Z 
2022-11-23T01:51:49.3014945Z Generating XML reports...
2022-11-23T01:51:49.3015395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014841.xml
2022-11-23T01:51:49.3015764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3015939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3016316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3016512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3016532Z 
2022-11-23T01:51:49.3016622Z Running tests...
2022-11-23T01:51:49.3016883Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3017244Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3017506Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3017723Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23967
2022-11-23T01:51:49.3017939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23968
2022-11-23T01:51:49.3018309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3018484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3018850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3019039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3019398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3019572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3019949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3020135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3020379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3020622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3021084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3021462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3021692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3021922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3022022Z ok (4.149s)
2022-11-23T01:51:49.3022041Z 
2022-11-23T01:51:49.3022304Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3022410Z Ran 1 test in 4.149s
2022-11-23T01:51:49.3022430Z 
2022-11-23T01:51:49.3022519Z OK
2022-11-23T01:51:49.3022539Z 
2022-11-23T01:51:49.3022658Z Generating XML reports...
2022-11-23T01:51:49.3023092Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014848.xml
2022-11-23T01:51:49.3023469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3023644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3024023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3024210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3024229Z 
2022-11-23T01:51:49.3024336Z Running tests...
2022-11-23T01:51:49.3024598Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3024907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3025163Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3025364Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24076
2022-11-23T01:51:49.3025585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24077
2022-11-23T01:51:49.3025955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3026125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3026548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3026743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3027368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3027546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3027922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3028102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3028346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3028589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3028992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3029388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3029616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3029842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3030176Z STAGE:2022-11-23 01:48:59 24077:24077 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3030585Z STAGE:2022-11-23 01:48:59 24076:24076 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3030918Z STAGE:2022-11-23 01:48:59 24076:24076 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3031242Z STAGE:2022-11-23 01:48:59 24077:24077 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3031591Z STAGE:2022-11-23 01:48:59 24076:24076 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3031936Z STAGE:2022-11-23 01:48:59 24077:24077 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3032259Z STAGE:2022-11-23 01:48:59 24076:24076 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3032574Z STAGE:2022-11-23 01:48:59 24077:24077 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3032894Z STAGE:2022-11-23 01:48:59 24077:24077 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3033448Z STAGE:2022-11-23 01:48:59 24077:24077 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:48:59 24076:24076 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3033470Z 
2022-11-23T01:51:49.3033819Z STAGE:2022-11-23 01:48:59 24076:24076 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3033904Z ok (4.217s)
2022-11-23T01:51:49.3033924Z 
2022-11-23T01:51:49.3034192Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3034300Z Ran 1 test in 4.218s
2022-11-23T01:51:49.3034319Z 
2022-11-23T01:51:49.3034409Z OK
2022-11-23T01:51:49.3034428Z 
2022-11-23T01:51:49.3034547Z Generating XML reports...
2022-11-23T01:51:49.3034998Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014855.xml
2022-11-23T01:51:49.3035368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3035549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3035926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3036171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3036193Z 
2022-11-23T01:51:49.3036307Z Running tests...
2022-11-23T01:51:49.3036569Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3036879Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3037130Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T01:51:49.3037149Z 
2022-11-23T01:51:49.3037407Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3037518Z Ran 1 test in 0.002s
2022-11-23T01:51:49.3037542Z 
2022-11-23T01:51:49.3037649Z OK (skipped=1)
2022-11-23T01:51:49.3037668Z 
2022-11-23T01:51:49.3037773Z Generating XML reports...
2022-11-23T01:51:49.3038225Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014901.xml
2022-11-23T01:51:49.3038597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3038772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3039149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3039338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3039357Z 
2022-11-23T01:51:49.3039464Z Running tests...
2022-11-23T01:51:49.3039725Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3040039Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3040350Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T01:51:49.3040371Z 
2022-11-23T01:51:49.3040629Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3040736Z Ran 1 test in 0.002s
2022-11-23T01:51:49.3040759Z 
2022-11-23T01:51:49.3040865Z OK (skipped=1)
2022-11-23T01:51:49.3040884Z 
2022-11-23T01:51:49.3041002Z Generating XML reports...
2022-11-23T01:51:49.3041451Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014904.xml
2022-11-23T01:51:49.3041813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3041988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3042364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3042544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3042563Z 
2022-11-23T01:51:49.3042669Z Running tests...
2022-11-23T01:51:49.3042927Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3043240Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3043497Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3043714Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24255
2022-11-23T01:51:49.3043932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24256
2022-11-23T01:51:49.3044301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3044459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3044840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3045022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3045452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3045633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3046004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3046189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3046434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3046678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3047067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3047459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3047690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3047917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3048156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.3048392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.3048787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.3049170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.3049570Z STAGE:2022-11-23 01:49:10 24255:24255 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3049875Z STAGE:2022-11-23 01:49:10 24256:24256 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3050208Z STAGE:2022-11-23 01:49:10 24256:24256 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3050531Z STAGE:2022-11-23 01:49:10 24255:24255 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3051097Z STAGE:2022-11-23 01:49:10 24255:24255 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:49:10 24256:24256 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3051118Z 
2022-11-23T01:51:49.3051640Z STAGE:2022-11-23 01:49:10 24255:24255 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:49:10 24256:24256 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3051664Z 
2022-11-23T01:51:49.3051990Z STAGE:2022-11-23 01:49:10 24256:24256 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3052314Z STAGE:2022-11-23 01:49:10 24255:24255 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3052660Z STAGE:2022-11-23 01:49:10 24256:24256 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3053005Z STAGE:2022-11-23 01:49:10 24255:24255 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3053106Z ok (4.270s)
2022-11-23T01:51:49.3053126Z 
2022-11-23T01:51:49.3053373Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3053483Z Ran 1 test in 4.271s
2022-11-23T01:51:49.3053503Z 
2022-11-23T01:51:49.3053593Z OK
2022-11-23T01:51:49.3053612Z 
2022-11-23T01:51:49.3053730Z Generating XML reports...
2022-11-23T01:51:49.3054184Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014906.xml
2022-11-23T01:51:49.3054555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3054730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3055159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3055358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3055378Z 
2022-11-23T01:51:49.3055468Z Running tests...
2022-11-23T01:51:49.3055729Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3056036Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3056287Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3056508Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24374
2022-11-23T01:51:49.3056724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24375
2022-11-23T01:51:49.3057089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3057268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3057630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3057822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3058185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3058358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3058729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3059031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3059274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3059519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3059925Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3060305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3060538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3060764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3060924Z skip: Skipped due to small world size. (4.262s)
2022-11-23T01:51:49.3060944Z 
2022-11-23T01:51:49.3061209Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3061320Z Ran 1 test in 4.263s
2022-11-23T01:51:49.3061339Z 
2022-11-23T01:51:49.3061443Z OK (skipped=1)
2022-11-23T01:51:49.3061462Z 
2022-11-23T01:51:49.3061584Z Generating XML reports...
2022-11-23T01:51:49.3062039Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014913.xml
2022-11-23T01:51:49.3062396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3062572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3062950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3063137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3063161Z 
2022-11-23T01:51:49.3063267Z Running tests...
2022-11-23T01:51:49.3063529Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3063837Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3064146Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3064356Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24483
2022-11-23T01:51:49.3064573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24484
2022-11-23T01:51:49.3064940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3065112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3065488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3065681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3066047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3066218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3066594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3066998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3067265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3067510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3067917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3068407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3068633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3068860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3068964Z ok (4.353s)
2022-11-23T01:51:49.3068985Z 
2022-11-23T01:51:49.3069249Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3069344Z Ran 1 test in 4.353s
2022-11-23T01:51:49.3069364Z 
2022-11-23T01:51:49.3069449Z OK
2022-11-23T01:51:49.3069468Z 
2022-11-23T01:51:49.3069590Z Generating XML reports...
2022-11-23T01:51:49.3070041Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014920.xml
2022-11-23T01:51:49.3070406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3070584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3070959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3071150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3071174Z 
2022-11-23T01:51:49.3071265Z Running tests...
2022-11-23T01:51:49.3071528Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3071839Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3072083Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3072297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24592
2022-11-23T01:51:49.3072513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24593
2022-11-23T01:51:49.3072887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3073061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3073424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3073692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3074074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3074245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3074618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3074803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3075044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3075293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3075691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3076072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3076299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3076522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3076624Z ok (4.366s)
2022-11-23T01:51:49.3076643Z 
2022-11-23T01:51:49.3076909Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3077019Z Ran 1 test in 4.366s
2022-11-23T01:51:49.3077038Z 
2022-11-23T01:51:49.3077184Z OK
2022-11-23T01:51:49.3077204Z 
2022-11-23T01:51:49.3077324Z Generating XML reports...
2022-11-23T01:51:49.3077780Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014927.xml
2022-11-23T01:51:49.3078131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3078305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3078679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3078869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3078889Z 
2022-11-23T01:51:49.3078994Z Running tests...
2022-11-23T01:51:49.3079252Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3079559Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3079825Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3080027Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24701
2022-11-23T01:51:49.3080241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24702
2022-11-23T01:51:49.3080613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3080787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3081163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3081349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3081711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3081887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3082268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3082440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3082734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3082984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3083385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3083776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3083999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3084229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3084329Z ok (4.259s)
2022-11-23T01:51:49.3084349Z 
2022-11-23T01:51:49.3084596Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3084706Z Ran 1 test in 4.259s
2022-11-23T01:51:49.3084726Z 
2022-11-23T01:51:49.3084817Z OK
2022-11-23T01:51:49.3084840Z 
2022-11-23T01:51:49.3084959Z Generating XML reports...
2022-11-23T01:51:49.3085409Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014934.xml
2022-11-23T01:51:49.3085776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3085947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3086323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3086570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3086590Z 
2022-11-23T01:51:49.3086681Z Running tests...
2022-11-23T01:51:49.3086947Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3087256Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3087547Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3087763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24810
2022-11-23T01:51:49.3087976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24811
2022-11-23T01:51:49.3088345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3088518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3088884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3089112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3089482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3089659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3090031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3090217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3090458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3090700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3091094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3091477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3091705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3092089Z STAGE:2022-11-23 01:49:44 24811:24811 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3092318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3092652Z STAGE:2022-11-23 01:49:45 24810:24810 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3092984Z STAGE:2022-11-23 01:49:45 24810:24810 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3093327Z STAGE:2022-11-23 01:49:45 24810:24810 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3093662Z STAGE:2022-11-23 01:49:45 24811:24811 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3094007Z STAGE:2022-11-23 01:49:45 24811:24811 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3094093Z ok (4.338s)
2022-11-23T01:51:49.3094113Z 
2022-11-23T01:51:49.3094375Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3094484Z Ran 1 test in 4.338s
2022-11-23T01:51:49.3094504Z 
2022-11-23T01:51:49.3094596Z OK
2022-11-23T01:51:49.3094616Z 
2022-11-23T01:51:49.3094739Z Generating XML reports...
2022-11-23T01:51:49.3095186Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014941.xml
2022-11-23T01:51:49.3095553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3095730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3096159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3096348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3096368Z 
2022-11-23T01:51:49.3096471Z Running tests...
2022-11-23T01:51:49.3096736Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3097047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3097327Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3097544Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24923
2022-11-23T01:51:49.3097758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24924
2022-11-23T01:51:49.3098130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3098293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3098670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3098855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3099218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3099391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3099766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3099948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3100186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3100418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3100813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3101255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3101489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3101824Z STAGE:2022-11-23 01:49:51 24924:24924 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3102049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3102376Z STAGE:2022-11-23 01:49:52 24923:24923 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3102710Z STAGE:2022-11-23 01:49:52 24923:24923 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3103270Z STAGE:2022-11-23 01:49:52 24924:24924 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:49:52 24923:24923 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3103290Z 
2022-11-23T01:51:49.3103638Z STAGE:2022-11-23 01:49:52 24924:24924 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3103724Z ok (4.323s)
2022-11-23T01:51:49.3103758Z 
2022-11-23T01:51:49.3104004Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3104111Z Ran 1 test in 4.323s
2022-11-23T01:51:49.3104131Z 
2022-11-23T01:51:49.3104221Z OK
2022-11-23T01:51:49.3104240Z 
2022-11-23T01:51:49.3104361Z Generating XML reports...
2022-11-23T01:51:49.3104810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014948.xml
2022-11-23T01:51:49.3105177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3105412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3105795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3105974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3105994Z 
2022-11-23T01:51:49.3106098Z Running tests...
2022-11-23T01:51:49.3106354Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3106660Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3107167Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3107393Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25036
2022-11-23T01:51:49.3107608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25037
2022-11-23T01:51:49.3107990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3108149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3108530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3108719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3109090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3109260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3109635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3109818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3110059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3110304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3110687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3111173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3111414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3111635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3111970Z STAGE:2022-11-23 01:49:58 25037:25037 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3112290Z STAGE:2022-11-23 01:49:58 25036:25036 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3112623Z STAGE:2022-11-23 01:49:58 25037:25037 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3112968Z STAGE:2022-11-23 01:49:58 25037:25037 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3113298Z STAGE:2022-11-23 01:49:58 25036:25036 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3113628Z STAGE:2022-11-23 01:49:58 25036:25036 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3113728Z ok (4.229s)
2022-11-23T01:51:49.3113749Z 
2022-11-23T01:51:49.3114005Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3114113Z Ran 1 test in 4.230s
2022-11-23T01:51:49.3114132Z 
2022-11-23T01:51:49.3114224Z OK
2022-11-23T01:51:49.3114244Z 
2022-11-23T01:51:49.3114363Z Generating XML reports...
2022-11-23T01:51:49.3114809Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014954.xml
2022-11-23T01:51:49.3115260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3115418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3115801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3115988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3116007Z 
2022-11-23T01:51:49.3116112Z Running tests...
2022-11-23T01:51:49.3116374Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3116680Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3116918Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T01:51:49.3116938Z 
2022-11-23T01:51:49.3117199Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3117309Z Ran 1 test in 0.002s
2022-11-23T01:51:49.3117329Z 
2022-11-23T01:51:49.3117419Z OK (skipped=1)
2022-11-23T01:51:49.3117438Z 
2022-11-23T01:51:49.3117556Z Generating XML reports...
2022-11-23T01:51:49.3118005Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015001.xml
2022-11-23T01:51:49.3118372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3118545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3118917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3119102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3119121Z 
2022-11-23T01:51:49.3119226Z Running tests...
2022-11-23T01:51:49.3119484Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3119779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3120040Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T01:51:49.3120061Z 
2022-11-23T01:51:49.3120368Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3120483Z Ran 1 test in 0.002s
2022-11-23T01:51:49.3120503Z 
2022-11-23T01:51:49.3120607Z OK (skipped=1)
2022-11-23T01:51:49.3120626Z 
2022-11-23T01:51:49.3120744Z Generating XML reports...
2022-11-23T01:51:49.3121195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015004.xml
2022-11-23T01:51:49.3121560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3121723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3122100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3122286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3122305Z 
2022-11-23T01:51:49.3122409Z Running tests...
2022-11-23T01:51:49.3122668Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3122979Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3123233Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T01:51:49.3123252Z 
2022-11-23T01:51:49.3123504Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3123613Z Ran 1 test in 0.002s
2022-11-23T01:51:49.3123633Z 
2022-11-23T01:51:49.3123722Z OK (skipped=1)
2022-11-23T01:51:49.3123799Z 
2022-11-23T01:51:49.3123926Z Generating XML reports...
2022-11-23T01:51:49.3124378Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015006.xml
2022-11-23T01:51:49.3124741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3124919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3125294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3125481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3125501Z 
2022-11-23T01:51:49.3125606Z Running tests...
2022-11-23T01:51:49.3125866Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3126158Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3126431Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3126649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25248
2022-11-23T01:51:49.3126864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25249
2022-11-23T01:51:49.3127237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3127412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3127788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3127976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3128330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3128503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3128882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3129063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3129360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3129610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3130013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3130405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3130634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3130949Z STAGE:2022-11-23 01:50:12 25249:25249 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3131176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3131506Z STAGE:2022-11-23 01:50:12 25248:25248 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3131845Z STAGE:2022-11-23 01:50:12 25249:25249 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3132188Z STAGE:2022-11-23 01:50:12 25249:25249 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3132517Z STAGE:2022-11-23 01:50:12 25248:25248 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3132862Z STAGE:2022-11-23 01:50:12 25248:25248 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3132963Z ok (4.327s)
2022-11-23T01:51:49.3132983Z 
2022-11-23T01:51:49.3133248Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3133401Z Ran 1 test in 4.327s
2022-11-23T01:51:49.3133421Z 
2022-11-23T01:51:49.3133509Z OK
2022-11-23T01:51:49.3133529Z 
2022-11-23T01:51:49.3133650Z Generating XML reports...
2022-11-23T01:51:49.3134104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015008.xml
2022-11-23T01:51:49.3134473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3134649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3135026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3135217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3135236Z 
2022-11-23T01:51:49.3135326Z Running tests...
2022-11-23T01:51:49.3135585Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3135897Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3136150Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3136364Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25361
2022-11-23T01:51:49.3136579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25362
2022-11-23T01:51:49.3136946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3137117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3137486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3137660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3138027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3138198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3138567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3138804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3139056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3139295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3139697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3140073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3140297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3140527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3140627Z ok (4.266s)
2022-11-23T01:51:49.3140646Z 
2022-11-23T01:51:49.3140903Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3141017Z Ran 1 test in 4.266s
2022-11-23T01:51:49.3141037Z 
2022-11-23T01:51:49.3141128Z OK
2022-11-23T01:51:49.3141148Z 
2022-11-23T01:51:49.3141268Z Generating XML reports...
2022-11-23T01:51:49.3141714Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015015.xml
2022-11-23T01:51:49.3142066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3142239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3142611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3142858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3142878Z 
2022-11-23T01:51:49.3142982Z Running tests...
2022-11-23T01:51:49.3143242Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3143558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3143842Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3144043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25470
2022-11-23T01:51:49.3144257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25471
2022-11-23T01:51:49.3144634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3144812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3145185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3145368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3145733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3145900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3146267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3146437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3146683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3147221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3147645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3148035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3148349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3148580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3148914Z STAGE:2022-11-23 01:50:26 25470:25470 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3149241Z STAGE:2022-11-23 01:50:26 25471:25471 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3149569Z STAGE:2022-11-23 01:50:26 25470:25470 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3149912Z STAGE:2022-11-23 01:50:26 25470:25470 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3150245Z STAGE:2022-11-23 01:50:26 25471:25471 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3150572Z STAGE:2022-11-23 01:50:26 25471:25471 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3150669Z ok (4.138s)
2022-11-23T01:51:49.3150693Z 
2022-11-23T01:51:49.3150953Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3151065Z Ran 1 test in 4.139s
2022-11-23T01:51:49.3151085Z 
2022-11-23T01:51:49.3151175Z OK
2022-11-23T01:51:49.3151194Z 
2022-11-23T01:51:49.3151312Z Generating XML reports...
2022-11-23T01:51:49.3151757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015022.xml
2022-11-23T01:51:49.3152122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3152357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3152738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3152923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3152942Z 
2022-11-23T01:51:49.3153048Z Running tests...
2022-11-23T01:51:49.3153311Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3153616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3153886Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3154097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25583
2022-11-23T01:51:49.3154310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25584
2022-11-23T01:51:49.3154670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3154843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3155216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3155407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3155768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3155939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3156311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3156494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3156722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3156968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3157366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3157801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3158033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3158366Z STAGE:2022-11-23 01:50:33 25584:25584 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3158592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3158912Z STAGE:2022-11-23 01:50:33 25583:25583 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T01:51:49.3159238Z STAGE:2022-11-23 01:50:33 25583:25583 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3159571Z STAGE:2022-11-23 01:50:33 25583:25583 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3159900Z STAGE:2022-11-23 01:50:33 25584:25584 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T01:51:49.3160243Z STAGE:2022-11-23 01:50:33 25584:25584 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T01:51:49.3160339Z ok (4.246s)
2022-11-23T01:51:49.3160359Z 
2022-11-23T01:51:49.3160618Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3160725Z Ran 1 test in 4.246s
2022-11-23T01:51:49.3160745Z 
2022-11-23T01:51:49.3160832Z OK
2022-11-23T01:51:49.3160851Z 
2022-11-23T01:51:49.3160971Z Generating XML reports...
2022-11-23T01:51:49.3161417Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015029.xml
2022-11-23T01:51:49.3161873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3162046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3162419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3162611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3162630Z 
2022-11-23T01:51:49.3162735Z Running tests...
2022-11-23T01:51:49.3162992Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3163299Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3163559Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3163761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25696
2022-11-23T01:51:49.3163980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25697
2022-11-23T01:51:49.3164344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3164514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3164890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3165077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3165438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3165609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3165979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3166156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3166393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3166632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3167084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3167490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3167715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3167939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3168037Z ok (4.342s)
2022-11-23T01:51:49.3168057Z 
2022-11-23T01:51:49.3168321Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3168421Z Ran 1 test in 4.343s
2022-11-23T01:51:49.3168441Z 
2022-11-23T01:51:49.3168530Z OK
2022-11-23T01:51:49.3168549Z 
2022-11-23T01:51:49.3168665Z Generating XML reports...
2022-11-23T01:51:49.3169112Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015035.xml
2022-11-23T01:51:49.3169479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3169648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3170019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3170204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3170224Z 
2022-11-23T01:51:49.3170314Z Running tests...
2022-11-23T01:51:49.3170568Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3170945Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3171213Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3171426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25880
2022-11-23T01:51:49.3171642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25881
2022-11-23T01:51:49.3172012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3172184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3172547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3172733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3173091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3173260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3173631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3173810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3174042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3174275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3174664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3175040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3175273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3175496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3175592Z ok (5.731s)
2022-11-23T01:51:49.3175612Z 
2022-11-23T01:51:49.3175869Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3176038Z Ran 1 test in 5.731s
2022-11-23T01:51:49.3176061Z 
2022-11-23T01:51:49.3176153Z OK
2022-11-23T01:51:49.3176172Z 
2022-11-23T01:51:49.3176294Z Generating XML reports...
2022-11-23T01:51:49.3176748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015042.xml
2022-11-23T01:51:49.3177100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3177272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3177648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3177839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3177859Z 
2022-11-23T01:51:49.3177964Z Running tests...
2022-11-23T01:51:49.3178223Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3178531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3178793Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3178992Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26081
2022-11-23T01:51:49.3179204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26082
2022-11-23T01:51:49.3179575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3179809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3180189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3180377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3180748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3180920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3181297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3181468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3181703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3181939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3182340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3182730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3182956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3183180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3183279Z ok (6.275s)
2022-11-23T01:51:49.3183299Z 
2022-11-23T01:51:49.3183561Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3183654Z Ran 1 test in 6.275s
2022-11-23T01:51:49.3183674Z 
2022-11-23T01:51:49.3183761Z OK
2022-11-23T01:51:49.3183781Z 
2022-11-23T01:51:49.3183899Z Generating XML reports...
2022-11-23T01:51:49.3184345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015051.xml
2022-11-23T01:51:49.3184717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3184888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3185318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3185513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3185533Z 
2022-11-23T01:51:49.3185623Z Running tests...
2022-11-23T01:51:49.3185884Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3186188Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3186448Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3186665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26196
2022-11-23T01:51:49.3187111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26197
2022-11-23T01:51:49.3187501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3187680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3188042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3188228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3188587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3188758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3189161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3189442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3189683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3189930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3190328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3190702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3190925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3191179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2x9ytygk
2022-11-23T01:51:49.3191444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2x9ytygk/_remote_module_non_scriptable.py
2022-11-23T01:51:49.3191671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3191921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2r9qu16d
2022-11-23T01:51:49.3192190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2r9qu16d/_remote_module_non_scriptable.py
2022-11-23T01:51:49.3192289Z ok (4.374s)
2022-11-23T01:51:49.3192309Z 
2022-11-23T01:51:49.3192574Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3192669Z Ran 1 test in 4.374s
2022-11-23T01:51:49.3192689Z 
2022-11-23T01:51:49.3192778Z OK
2022-11-23T01:51:49.3192797Z 
2022-11-23T01:51:49.3192916Z Generating XML reports...
2022-11-23T01:51:49.3193366Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015100.xml
2022-11-23T01:51:49.3193733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3193912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3194289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3194544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3194567Z 
2022-11-23T01:51:49.3194665Z Running tests...
2022-11-23T01:51:49.3194930Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3195237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3195486Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3195703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26309
2022-11-23T01:51:49.3195918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26310
2022-11-23T01:51:49.3196293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3196462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3196843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3197016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3197375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3197541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3197908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3198094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3198397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3198636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3199040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3199421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3199650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3199874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3200125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjh2qkd0a
2022-11-23T01:51:49.3200391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjh2qkd0a/_remote_module_non_scriptable.py
2022-11-23T01:51:49.3200641Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_3yvso5
2022-11-23T01:51:49.3200900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_3yvso5/_remote_module_non_scriptable.py
2022-11-23T01:51:49.3200996Z ok (5.733s)
2022-11-23T01:51:49.3201016Z 
2022-11-23T01:51:49.3201286Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3201381Z Ran 1 test in 5.733s
2022-11-23T01:51:49.3201400Z 
2022-11-23T01:51:49.3201486Z OK
2022-11-23T01:51:49.3201506Z 
2022-11-23T01:51:49.3201622Z Generating XML reports...
2022-11-23T01:51:49.3202069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015106.xml
2022-11-23T01:51:49.3202434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3202602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3202983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3203171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3203190Z 
2022-11-23T01:51:49.3203293Z Running tests...
2022-11-23T01:51:49.3203648Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3203971Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3204259Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3204468Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26420
2022-11-23T01:51:49.3204679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26421
2022-11-23T01:51:49.3205045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3205223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3205594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3205771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3206134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3206303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3206673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3206852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3207091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3207389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3207791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3208187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3208401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3208621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3208872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyablcgek
2022-11-23T01:51:49.3209137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyablcgek/_remote_module_non_scriptable.py
2022-11-23T01:51:49.3209386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwidngb0
2022-11-23T01:51:49.3209656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwidngb0/_remote_module_non_scriptable.py
2022-11-23T01:51:49.3210437Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:51:49.3211207Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T01:51:49.3211309Z ok (6.256s)
2022-11-23T01:51:49.3211329Z 
2022-11-23T01:51:49.3211648Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3211763Z Ran 1 test in 6.256s
2022-11-23T01:51:49.3211783Z 
2022-11-23T01:51:49.3211870Z OK
2022-11-23T01:51:49.3211889Z 
2022-11-23T01:51:49.3211994Z Generating XML reports...
2022-11-23T01:51:49.3212446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015115.xml
2022-11-23T01:51:49.3212811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3212981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3213366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3213553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3213572Z 
2022-11-23T01:51:49.3213672Z Running tests...
2022-11-23T01:51:49.3213934Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3214229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3214511Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3214724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26535
2022-11-23T01:51:49.3214933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26536
2022-11-23T01:51:49.3215299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3215528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3215909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3216097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3216459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3216616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3216988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3217168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3217408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3217648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3218045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3218433Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3218662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3218885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3219109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.3219342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.3219730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.3220120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.3220355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:51:49.3220652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:51:49.3221053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.3221437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.3221694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph12hx1il
2022-11-23T01:51:49.3221946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph12hx1il/_remote_module_non_scriptable.py
2022-11-23T01:51:49.3222199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ashsuem
2022-11-23T01:51:49.3222465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ashsuem/_remote_module_non_scriptable.py
2022-11-23T01:51:49.3222561Z ok (10.849s)
2022-11-23T01:51:49.3222582Z 
2022-11-23T01:51:49.3222844Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3222957Z Ran 1 test in 10.849s
2022-11-23T01:51:49.3222977Z 
2022-11-23T01:51:49.3223067Z OK
2022-11-23T01:51:49.3223087Z 
2022-11-23T01:51:49.3223208Z Generating XML reports...
2022-11-23T01:51:49.3223654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015124.xml
2022-11-23T01:51:49.3224008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3224177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3224551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3224799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3224819Z 
2022-11-23T01:51:49.3224920Z Running tests...
2022-11-23T01:51:49.3225185Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3225497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T01:51:49.3225777Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T01:51:49.3225978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26658
2022-11-23T01:51:49.3226191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26659
2022-11-23T01:51:49.3226563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3226735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3227370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3227561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3227932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T01:51:49.3228102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T01:51:49.3228477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T01:51:49.3228650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T01:51:49.3228892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T01:51:49.3229130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T01:51:49.3229529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3229920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T01:51:49.3230227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T01:51:49.3230462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T01:51:49.3230701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T01:51:49.3230939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T01:51:49.3231322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.3231714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T01:51:49.3231953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T01:51:49.3232189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T01:51:49.3232573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.3232958Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T01:51:49.3233210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fxlmagd
2022-11-23T01:51:49.3233479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fxlmagd/_remote_module_non_scriptable.py
2022-11-23T01:51:49.3233727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dsrxd0v
2022-11-23T01:51:49.3234056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dsrxd0v/_remote_module_non_scriptable.py
2022-11-23T01:51:49.3234154Z ok (10.974s)
2022-11-23T01:51:49.3234175Z 
2022-11-23T01:51:49.3234442Z ----------------------------------------------------------------------
2022-11-23T01:51:49.3234557Z Ran 1 test in 10.974s
2022-11-23T01:51:49.3234576Z 
2022-11-23T01:51:49.3234665Z OK
2022-11-23T01:51:49.3234684Z 
2022-11-23T01:51:49.3234803Z Generating XML reports...
2022-11-23T01:51:49.3235251Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015137.xml
2022-11-23T01:51:49.3235271Z 
2022-11-23T01:51:49.3235990Z ##[endgroup]
2022-11-23T01:51:49.3236461Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_oa_8p1ar)
2022-11-23T01:51:49.3236481Z 
2022-11-23T01:51:49.3236703Z Running distributed tests for the gloo backend with file init_method in shard 1 of 3
2022-11-23T01:51:49.3237211Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 01:51:49.062256]
2022-11-23T02:21:02.3839174Z 
2022-11-23T02:21:02.3840553Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-11-23T02:21:02.3844870Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_pvjooool)
2022-11-23T02:21:02.3845740Z <unittest.suite.TestSuite tests=[]>
2022-11-23T02:21:02.3909828Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-11-23T02:21:02.3970602Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3971496Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3972250Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3972968Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3976793Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3977631Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3978507Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3979396Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3980284Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3981249Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3982235Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3983164Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3984093Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3985003Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3985855Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3987238Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3988106Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3988890Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3989649Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3990421Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3991282Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3992122Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3992829Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3993535Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3994290Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3995050Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3995754Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3996513Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3997232Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3997882Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3998596Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.3999297Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4000140Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4000847Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4001585Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4002310Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4003017Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4003759Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4004486Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4005195Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4005901Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4006676Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4007472Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4008246Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4009016Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4009794Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4010583Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4011329Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4012079Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4012867Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4013626Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4014348Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4015073Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4015833Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4016566Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4017269Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4018109Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4018839Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4019554Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4020244Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4020908Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4021621Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4022296Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4022999Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4023679Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4024356Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4025099Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4025791Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4026422Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4027624Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4028323Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4029033Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4029796Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4030552Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4031202Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4032138Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4032893Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4033636Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4034432Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4035239Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4036021Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4036766Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4037567Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4038428Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4039225Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4040093Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4040978Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4041866Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4042682Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4043516Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4044334Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4045144Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4045995Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4046854Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4047735Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4048502Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4049325Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4050133Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4051042Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4051783Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4052492Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4053158Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4053794Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4054325Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4054724Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4055097Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4055488Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4055887Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4056278Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4056937Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4057585Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4058004Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4058415Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4058833Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4059243Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4059645Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4060179Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4060610Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4061017Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4061429Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4061821Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4062192Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4062564Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4062954Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4063341Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4063717Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4064171Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4064694Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4065145Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4065541Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4065959Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4066391Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4067448Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4067986Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4068432Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4068845Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4069287Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4069704Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4070076Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4070450Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4070964Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4071395Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4071810Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4072249Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4072666Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4073079Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4073521Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4074023Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4074585Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4075186Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4075786Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4076377Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4076981Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4077667Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4078247Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4078847Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4079392Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4079891Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4080317Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4080700Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4081094Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4081490Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4081886Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4082305Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4082726Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4083188Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4083654Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4084067Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4084438Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4084848Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4085283Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4085698Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4086110Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4086523Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4086953Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4087416Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4087849Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4088260Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4088651Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4089069Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4089469Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4089875Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4090339Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4090776Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4091174Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4091541Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4091939Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4092356Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4092753Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4093133Z test_gather (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4093495Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4093844Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4094221Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4094661Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4095032Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4095401Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4095783Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4096148Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4096486Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4096863Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4097254Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4097627Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4097992Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4098334Z test_isend (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4098703Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4099117Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4099527Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4099960Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4100410Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4100820Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4101218Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4101646Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4102068Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4102471Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4102887Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4103301Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4103706Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4104096Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4104486Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4104933Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4105392Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4105881Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4106344Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4107209Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4107766Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4108227Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4108656Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4109057Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4109487Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4109923Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4110372Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4110834Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4111342Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4111823Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4112322Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4112713Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4113115Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4113520Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4113891Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4114269Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4114655Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4115026Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4115397Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4115755Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4116112Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4116492Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4116884Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4117268Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4117642Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4118012Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4118397Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4118764Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4119124Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4119486Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4119845Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4120217Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4120596Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4120972Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4121345Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4121724Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4122088Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4122525Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4122959Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4123403Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4123810Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4124204Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4124604Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4125011Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4125428Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4125820Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4126229Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4126643Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4127058Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4127456Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4127842Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4128237Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4128613Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4129030Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4129462Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4129972Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4130706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4131140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4131723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4132187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4132419Z 
2022-11-23T02:21:02.4132528Z Running tests...
2022-11-23T02:21:02.4132917Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4133444Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4134032Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4134580Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26814
2022-11-23T02:21:02.4135028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26815
2022-11-23T02:21:02.4135634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4136080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4136635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4137095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4137668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4138106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4138658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4139114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4139562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4140094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4140763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4141455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4141980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4142434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4142955Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:21:02.4143846Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:21:02.4144520Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:21:02.4145321Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:21:02.4145972Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:21:02.4147212Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:21:02.4148053Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:21:02.4148875Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:21:02.4149533Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:21:02.4150344Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:21:02.4151002Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:21:02.4151799Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:21:02.4152465Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:21:02.4153278Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:21:02.4153929Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-11-23T02:21:02.4154722Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-11-23T02:21:02.4155202Z ok (6.866s)
2022-11-23T02:21:02.4155352Z 
2022-11-23T02:21:02.4155618Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4155947Z Ran 1 test in 6.867s
2022-11-23T02:21:02.4156089Z 
2022-11-23T02:21:02.4156187Z OK
2022-11-23T02:21:02.4156320Z 
2022-11-23T02:21:02.4156441Z Generating XML reports...
2022-11-23T02:21:02.4157050Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015153.xml
2022-11-23T02:21:02.4157792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4158324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4158913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4159384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4159614Z 
2022-11-23T02:21:02.4159703Z Running tests...
2022-11-23T02:21:02.4160100Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4160625Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4161142Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-11-23T02:21:02.4161452Z 
2022-11-23T02:21:02.4161720Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4162040Z Ran 1 test in 0.004s
2022-11-23T02:21:02.4162199Z 
2022-11-23T02:21:02.4162308Z OK (skipped=1)
2022-11-23T02:21:02.4162462Z 
2022-11-23T02:21:02.4162567Z Generating XML reports...
2022-11-23T02:21:02.4163169Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015202.xml
2022-11-23T02:21:02.4163880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4164325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4164881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4165415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4165646Z 
2022-11-23T02:21:02.4165752Z Running tests...
2022-11-23T02:21:02.4166137Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4166670Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4167178Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4167660Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26959
2022-11-23T02:21:02.4168089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26960
2022-11-23T02:21:02.4168689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4169133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4169692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4170161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4170732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4171176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4171728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4172183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4172632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4173110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4173765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4174452Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4174978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4175489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4175837Z ok (4.244s)
2022-11-23T02:21:02.4175984Z 
2022-11-23T02:21:02.4176254Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4176577Z Ran 1 test in 4.244s
2022-11-23T02:21:02.4176718Z 
2022-11-23T02:21:02.4176808Z OK
2022-11-23T02:21:02.4176938Z 
2022-11-23T02:21:02.4177059Z Generating XML reports...
2022-11-23T02:21:02.4177662Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015205.xml
2022-11-23T02:21:02.4178365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4178809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4179375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4179846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4180058Z 
2022-11-23T02:21:02.4180164Z Running tests...
2022-11-23T02:21:02.4180557Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4181082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4181593Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4182647Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.613s)
2022-11-23T02:21:02.4183230Z 
2022-11-23T02:21:02.4183496Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4183823Z Ran 1 test in 1.613s
2022-11-23T02:21:02.4183983Z 
2022-11-23T02:21:02.4184091Z OK (skipped=1)
2022-11-23T02:21:02.4184227Z 
2022-11-23T02:21:02.4184349Z Generating XML reports...
2022-11-23T02:21:02.4184950Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015212.xml
2022-11-23T02:21:02.4185656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4186088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4186659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4187602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4187835Z 
2022-11-23T02:21:02.4187943Z Running tests...
2022-11-23T02:21:02.4188339Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4188873Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4189406Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4189920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27102
2022-11-23T02:21:02.4190353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27103
2022-11-23T02:21:02.4190956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4191410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4191965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4192428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4193091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4193546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4194103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4194565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4195016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4195495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4196152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4196838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4197365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4197851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7kzi2b1
2022-11-23T02:21:02.4198385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7kzi2b1/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4198890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4199384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sjb29q1
2022-11-23T02:21:02.4199897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sjb29q1/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4200491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4200976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4201307Z ok (4.315s)
2022-11-23T02:21:02.4201451Z 
2022-11-23T02:21:02.4201726Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4202055Z Ran 1 test in 4.316s
2022-11-23T02:21:02.4202214Z 
2022-11-23T02:21:02.4202306Z OK
2022-11-23T02:21:02.4202420Z 
2022-11-23T02:21:02.4202544Z Generating XML reports...
2022-11-23T02:21:02.4203146Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015216.xml
2022-11-23T02:21:02.4203852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4204287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4204867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4205330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4205558Z 
2022-11-23T02:21:02.4205666Z Running tests...
2022-11-23T02:21:02.4206048Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4206575Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4207124Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4207634Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27215
2022-11-23T02:21:02.4208082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27216
2022-11-23T02:21:02.4208679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4209127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4209671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4210170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4210753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4211200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4211777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4212239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4212688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4213171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4213824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4214514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4215031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4215481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4215967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17ffs5ex
2022-11-23T02:21:02.4216487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgkoz2gv
2022-11-23T02:21:02.4217002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17ffs5ex/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4217610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgkoz2gv/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4218125Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4218612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4218943Z ok (4.204s)
2022-11-23T02:21:02.4219086Z 
2022-11-23T02:21:02.4219360Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4219687Z Ran 1 test in 4.204s
2022-11-23T02:21:02.4219846Z 
2022-11-23T02:21:02.4219919Z OK
2022-11-23T02:21:02.4220052Z 
2022-11-23T02:21:02.4220175Z Generating XML reports...
2022-11-23T02:21:02.4220781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015223.xml
2022-11-23T02:21:02.4221496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4221931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4222503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4222965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4223195Z 
2022-11-23T02:21:02.4223306Z Running tests...
2022-11-23T02:21:02.4223686Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4224212Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4224753Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4225261Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27328
2022-11-23T02:21:02.4225706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27329
2022-11-23T02:21:02.4226313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4227086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4227875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4228354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4228935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4229357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4229926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4230382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4230839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4231314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4231966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4232656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4233178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4233629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4234115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp344j69ir
2022-11-23T02:21:02.4234649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp344j69ir/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4235238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpszu6bdqu
2022-11-23T02:21:02.4235775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpszu6bdqu/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4236288Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4236775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4237237Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4237713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4238178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4238628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4239098Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4239567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4240031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4240484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4240949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4241418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4241758Z ok (6.945s)
2022-11-23T02:21:02.4241888Z 
2022-11-23T02:21:02.4242172Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4242498Z Ran 1 test in 6.945s
2022-11-23T02:21:02.4242658Z 
2022-11-23T02:21:02.4242749Z OK
2022-11-23T02:21:02.4242863Z 
2022-11-23T02:21:02.4242984Z Generating XML reports...
2022-11-23T02:21:02.4243595Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015229.xml
2022-11-23T02:21:02.4244306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4244736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4245357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4245833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4246062Z 
2022-11-23T02:21:02.4246166Z Running tests...
2022-11-23T02:21:02.4246552Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4247079Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4247642Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4248183Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27443
2022-11-23T02:21:02.4248614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27444
2022-11-23T02:21:02.4249217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4249667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4250222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4250686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4251259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4251700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4252319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4252780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4253231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4253714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4254365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4255047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4255572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4256025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4256522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6d0uwj3g
2022-11-23T02:21:02.4257058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6d0uwj3g/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4257633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpewra987g
2022-11-23T02:21:02.4258152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpewra987g/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4258667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4259146Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4259605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4260079Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4260425Z ok (5.829s)
2022-11-23T02:21:02.4260571Z 
2022-11-23T02:21:02.4260846Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4261156Z Ran 1 test in 5.829s
2022-11-23T02:21:02.4261314Z 
2022-11-23T02:21:02.4261406Z OK
2022-11-23T02:21:02.4261538Z 
2022-11-23T02:21:02.4261658Z Generating XML reports...
2022-11-23T02:21:02.4262304Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015239.xml
2022-11-23T02:21:02.4263034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4263484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4264057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4264505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4264731Z 
2022-11-23T02:21:02.4264844Z Running tests...
2022-11-23T02:21:02.4265243Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4265751Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4266320Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4267304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27558
2022-11-23T02:21:02.4267923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27559
2022-11-23T02:21:02.4268534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4268982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4269549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4270136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4270702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4271141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4271709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4272151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4272603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4273095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4273747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4274427Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4274944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4275410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4275910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdosl0ga9
2022-11-23T02:21:02.4276430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdosl0ga9/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4276957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7242qp1
2022-11-23T02:21:02.4277490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7242qp1/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4277993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4278468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4278941Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4279411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4279930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4280411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4280880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4281330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4281678Z ok (5.916s)
2022-11-23T02:21:02.4281826Z 
2022-11-23T02:21:02.4282101Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4282430Z Ran 1 test in 5.916s
2022-11-23T02:21:02.4282578Z 
2022-11-23T02:21:02.4282670Z OK
2022-11-23T02:21:02.4282798Z 
2022-11-23T02:21:02.4282918Z Generating XML reports...
2022-11-23T02:21:02.4283528Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015247.xml
2022-11-23T02:21:02.4284231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4284680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4285252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4285716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4285930Z 
2022-11-23T02:21:02.4286036Z Running tests...
2022-11-23T02:21:02.4286433Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4286957Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4287602Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4288165Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27673
2022-11-23T02:21:02.4288613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27674
2022-11-23T02:21:02.4289224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4289654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4290218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4290680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4291249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4291676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4292246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4292711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4293149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4293644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4294303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4294987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4295492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4295965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4296462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcqfcs9jw
2022-11-23T02:21:02.4297052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcqfcs9jw/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4297572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1x1v8d4t
2022-11-23T02:21:02.4298105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1x1v8d4t/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4298617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4299084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4299431Z ok (6.024s)
2022-11-23T02:21:02.4299576Z 
2022-11-23T02:21:02.4299853Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4300189Z Ran 1 test in 6.024s
2022-11-23T02:21:02.4300330Z 
2022-11-23T02:21:02.4300421Z OK
2022-11-23T02:21:02.4300554Z 
2022-11-23T02:21:02.4300677Z Generating XML reports...
2022-11-23T02:21:02.4301291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015255.xml
2022-11-23T02:21:02.4301988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4302435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4303009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4303476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4303688Z 
2022-11-23T02:21:02.4303796Z Running tests...
2022-11-23T02:21:02.4304191Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4304785Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4305350Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4305911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27788
2022-11-23T02:21:02.4306361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27789
2022-11-23T02:21:02.4307372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4307820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4308404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4308874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4309460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4309881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4310449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4310907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4311337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4311829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4312481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4313164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4313667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4314138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4314758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoh5wp4_l
2022-11-23T02:21:02.4315317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoh5wp4_l/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4315828Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn3t2nis7
2022-11-23T02:21:02.4316359Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn3t2nis7/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4316869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4317332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4317809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4318284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4318624Z ok (6.552s)
2022-11-23T02:21:02.4318754Z 
2022-11-23T02:21:02.4319036Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4319364Z Ran 1 test in 6.552s
2022-11-23T02:21:02.4319525Z 
2022-11-23T02:21:02.4319618Z OK
2022-11-23T02:21:02.4319750Z 
2022-11-23T02:21:02.4319854Z Generating XML reports...
2022-11-23T02:21:02.4320456Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015304.xml
2022-11-23T02:21:02.4321167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4321616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4322253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4322718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4322950Z 
2022-11-23T02:21:02.4323058Z Running tests...
2022-11-23T02:21:02.4323445Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4323969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4324527Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4325060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27903
2022-11-23T02:21:02.4325488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27904
2022-11-23T02:21:02.4326084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4326532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4327084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4327551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4328131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4328566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4329114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4329575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4330028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4330527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4331165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4331901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4332431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4332879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4333374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47knujjd
2022-11-23T02:21:02.4333912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47knujjd/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4334438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4xhb3a1l
2022-11-23T02:21:02.4334959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4xhb3a1l/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4335471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4335950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4336433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4336889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4337228Z ok (6.346s)
2022-11-23T02:21:02.4337374Z 
2022-11-23T02:21:02.4337645Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4337954Z Ran 1 test in 6.347s
2022-11-23T02:21:02.4338111Z 
2022-11-23T02:21:02.4338203Z OK
2022-11-23T02:21:02.4338336Z 
2022-11-23T02:21:02.4338456Z Generating XML reports...
2022-11-23T02:21:02.4339043Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015313.xml
2022-11-23T02:21:02.4339819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4340268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4340844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4341292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4341522Z 
2022-11-23T02:21:02.4341628Z Running tests...
2022-11-23T02:21:02.4342025Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4342537Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4343122Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4343688Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28018
2022-11-23T02:21:02.4344134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28019
2022-11-23T02:21:02.4344727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4345173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4345749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4346217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4347025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4347583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4348170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4348622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4349076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4349655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4350330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4351001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4351521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4351990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4352494Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpai891ok2
2022-11-23T02:21:02.4353014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpai891ok2/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4353536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qey__9_
2022-11-23T02:21:02.4354061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qey__9_/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4354549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4355028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4355502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4355980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4356304Z ok (5.846s)
2022-11-23T02:21:02.4356526Z 
2022-11-23T02:21:02.4356798Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4357123Z Ran 1 test in 5.846s
2022-11-23T02:21:02.4357325Z 
2022-11-23T02:21:02.4357399Z OK
2022-11-23T02:21:02.4357532Z 
2022-11-23T02:21:02.4357655Z Generating XML reports...
2022-11-23T02:21:02.4358268Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015322.xml
2022-11-23T02:21:02.4358986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4359417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4359989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4360452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4360678Z 
2022-11-23T02:21:02.4360784Z Running tests...
2022-11-23T02:21:02.4361170Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4361697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4362246Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4363298Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.628s)
2022-11-23T02:21:02.4363813Z 
2022-11-23T02:21:02.4364080Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4364404Z Ran 1 test in 1.628s
2022-11-23T02:21:02.4364563Z 
2022-11-23T02:21:02.4364670Z OK (skipped=1)
2022-11-23T02:21:02.4364824Z 
2022-11-23T02:21:02.4364932Z Generating XML reports...
2022-11-23T02:21:02.4365537Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015330.xml
2022-11-23T02:21:02.4366245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4366750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4367320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4367787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4368018Z 
2022-11-23T02:21:02.4368123Z Running tests...
2022-11-23T02:21:02.4368505Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4369029Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4369570Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4370100Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28167
2022-11-23T02:21:02.4370529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28168
2022-11-23T02:21:02.4371133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4371582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4372137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4372608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4373183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4373622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4374235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4374697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4375147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4375645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4376283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4376965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4377485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4377939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4378286Z ok (4.320s)
2022-11-23T02:21:02.4378431Z 
2022-11-23T02:21:02.4378697Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4379023Z Ran 1 test in 4.320s
2022-11-23T02:21:02.4379164Z 
2022-11-23T02:21:02.4379255Z OK
2022-11-23T02:21:02.4379386Z 
2022-11-23T02:21:02.4379511Z Generating XML reports...
2022-11-23T02:21:02.4380118Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015335.xml
2022-11-23T02:21:02.4380818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4381266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4381837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4382300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4382533Z 
2022-11-23T02:21:02.4382623Z Running tests...
2022-11-23T02:21:02.4383022Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4383543Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4384134Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4385208Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.636s)
2022-11-23T02:21:02.4385729Z 
2022-11-23T02:21:02.4385993Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4386315Z Ran 1 test in 1.636s
2022-11-23T02:21:02.4386479Z 
2022-11-23T02:21:02.4386587Z OK (skipped=1)
2022-11-23T02:21:02.4386723Z 
2022-11-23T02:21:02.4387168Z Generating XML reports...
2022-11-23T02:21:02.4387914Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015341.xml
2022-11-23T02:21:02.4388638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4389071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4389643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4390105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4390334Z 
2022-11-23T02:21:02.4390439Z Running tests...
2022-11-23T02:21:02.4390818Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4391342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4391995Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4392482Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28310
2022-11-23T02:21:02.4392934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28311
2022-11-23T02:21:02.4393540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4393992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4394545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4395008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4395579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4396026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4396575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4397035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4397488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4397973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4398628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4399310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4399828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4400286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4400616Z ok (5.746s)
2022-11-23T02:21:02.4400763Z 
2022-11-23T02:21:02.4401030Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4401335Z Ran 1 test in 5.747s
2022-11-23T02:21:02.4401566Z 
2022-11-23T02:21:02.4401664Z OK
2022-11-23T02:21:02.4401799Z 
2022-11-23T02:21:02.4401922Z Generating XML reports...
2022-11-23T02:21:02.4402529Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015346.xml
2022-11-23T02:21:02.4403227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4403674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4404244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4404696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4404923Z 
2022-11-23T02:21:02.4405030Z Running tests...
2022-11-23T02:21:02.4405428Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4405955Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4406439Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-11-23T02:21:02.4406723Z 
2022-11-23T02:21:02.4406982Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4407301Z Ran 1 test in 0.002s
2022-11-23T02:21:02.4407460Z 
2022-11-23T02:21:02.4407549Z OK (skipped=1)
2022-11-23T02:21:02.4407704Z 
2022-11-23T02:21:02.4407825Z Generating XML reports...
2022-11-23T02:21:02.4408421Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015354.xml
2022-11-23T02:21:02.4409189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4409621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4410198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4410660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4410889Z 
2022-11-23T02:21:02.4410994Z Running tests...
2022-11-23T02:21:02.4411370Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4411894Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4412348Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4412804Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4413285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28454
2022-11-23T02:21:02.4413726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28455
2022-11-23T02:21:02.4414335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4414764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4415334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4415804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4416360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4416797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4417358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4417822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4418254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4418797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4419464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4420156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4420660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4421125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4421626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xfrruyx
2022-11-23T02:21:02.4422125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtmobw1t
2022-11-23T02:21:02.4422664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xfrruyx/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4423208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtmobw1t/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4423587Z ok (4.288s)
2022-11-23T02:21:02.4423717Z 
2022-11-23T02:21:02.4423988Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4424310Z Ran 1 test in 4.289s
2022-11-23T02:21:02.4424470Z 
2022-11-23T02:21:02.4424561Z OK
2022-11-23T02:21:02.4424694Z 
2022-11-23T02:21:02.4424798Z Generating XML reports...
2022-11-23T02:21:02.4425399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015356.xml
2022-11-23T02:21:02.4426179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4426629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4427537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4428019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4428248Z 
2022-11-23T02:21:02.4428355Z Running tests...
2022-11-23T02:21:02.4428738Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4429265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4429741Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4430239Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4430711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28567
2022-11-23T02:21:02.4431156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28568
2022-11-23T02:21:02.4431757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4432192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4432763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4433227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4433808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4434230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4434796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4435259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4435709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4468923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4469736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4470421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4470949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4471453Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2o9tbuv1
2022-11-23T02:21:02.4471983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2o9tbuv1/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4472501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4472998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1v44cqg4
2022-11-23T02:21:02.4473533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1v44cqg4/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4474031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4474517Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4474867Z ok (4.338s)
2022-11-23T02:21:02.4475016Z 
2022-11-23T02:21:02.4475275Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4475603Z Ran 1 test in 4.338s
2022-11-23T02:21:02.4475764Z 
2022-11-23T02:21:02.4475856Z OK
2022-11-23T02:21:02.4475991Z 
2022-11-23T02:21:02.4476208Z Generating XML reports...
2022-11-23T02:21:02.4476810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015403.xml
2022-11-23T02:21:02.4477527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4477979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4478537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4479005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4479234Z 
2022-11-23T02:21:02.4479341Z Running tests...
2022-11-23T02:21:02.4479745Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4480255Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4480752Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4481288Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4481792Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28680
2022-11-23T02:21:02.4482228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28681
2022-11-23T02:21:02.4482836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4483280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4483841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4484302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4484877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4485317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4485868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4486327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4486833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4487323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4487987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4488676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4489197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4489686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpavb0z24y
2022-11-23T02:21:02.4490219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpavb0z24y/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4490727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4491226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_0kxci5
2022-11-23T02:21:02.4491736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_0kxci5/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4492244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4492723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.4493057Z ok (4.339s)
2022-11-23T02:21:02.4493202Z 
2022-11-23T02:21:02.4493545Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4493871Z Ran 1 test in 4.339s
2022-11-23T02:21:02.4494032Z 
2022-11-23T02:21:02.4494123Z OK
2022-11-23T02:21:02.4494238Z 
2022-11-23T02:21:02.4494360Z Generating XML reports...
2022-11-23T02:21:02.4494972Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015410.xml
2022-11-23T02:21:02.4495686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4496117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4496689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4497156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4497384Z 
2022-11-23T02:21:02.4497490Z Running tests...
2022-11-23T02:21:02.4497874Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4498408Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4498880Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-11-23T02:21:02.4499361Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4499838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28793
2022-11-23T02:21:02.4500286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28794
2022-11-23T02:21:02.4500891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4501317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4501889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4502356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4502917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4503358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4503988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4504458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4504892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4505387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4506043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4506739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4507510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4507990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4508492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6lwhnutq
2022-11-23T02:21:02.4509000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpir557241
2022-11-23T02:21:02.4509537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6lwhnutq/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4510075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpir557241/_remote_module_non_scriptable.py
2022-11-23T02:21:02.4510450Z ok (4.258s)
2022-11-23T02:21:02.4510580Z 
2022-11-23T02:21:02.4510858Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4511288Z Ran 1 test in 4.258s
2022-11-23T02:21:02.4511453Z 
2022-11-23T02:21:02.4511548Z OK
2022-11-23T02:21:02.4511681Z 
2022-11-23T02:21:02.4511785Z Generating XML reports...
2022-11-23T02:21:02.4512395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015417.xml
2022-11-23T02:21:02.4513116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4513566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4514124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4514591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4514818Z 
2022-11-23T02:21:02.4514925Z Running tests...
2022-11-23T02:21:02.4515311Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4515842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4516340Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4516817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28906
2022-11-23T02:21:02.4517253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28907
2022-11-23T02:21:02.4517858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4518308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4518885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4519334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4519906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4520349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4520898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4521435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4521897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4522401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4523041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4523727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4524252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4524722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4525287Z STAGE:2022-11-23 01:54:27 28906:28906 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4525861Z STAGE:2022-11-23 01:54:27 28907:28907 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4526650Z STAGE:2022-11-23 01:54:27 28907:28907 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:54:27 28906:28906 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4527034Z 
2022-11-23T02:21:02.4527601Z STAGE:2022-11-23 01:54:27 28906:28906 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:54:27 28907:28907 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4528004Z 
2022-11-23T02:21:02.4528381Z STAGE:2022-11-23 01:54:27 28906:28906 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4528949Z STAGE:2022-11-23 01:54:27 28907:28907 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4529532Z STAGE:2022-11-23 01:54:27 28907:28907 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4530108Z STAGE:2022-11-23 01:54:27 28906:28906 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4530677Z STAGE:2022-11-23 01:54:27 28907:28907 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4531276Z STAGE:2022-11-23 01:54:27 28906:28906 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4531620Z ok (4.247s)
2022-11-23T02:21:02.4531767Z 
2022-11-23T02:21:02.4532016Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4532338Z Ran 1 test in 4.247s
2022-11-23T02:21:02.4532501Z 
2022-11-23T02:21:02.4532592Z OK
2022-11-23T02:21:02.4532723Z 
2022-11-23T02:21:02.4532846Z Generating XML reports...
2022-11-23T02:21:02.4533436Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015424.xml
2022-11-23T02:21:02.4534149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4534597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4535152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4535626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4535852Z 
2022-11-23T02:21:02.4535957Z Running tests...
2022-11-23T02:21:02.4536225Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4536535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4536817Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4537018Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29019
2022-11-23T02:21:02.4537234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29020
2022-11-23T02:21:02.4537666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4537847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4538223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4538396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4538775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4538969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4539325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4539512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4539760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4540004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4540402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4540795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4541027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4541442Z STAGE:2022-11-23 01:54:34 29020:29020 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4542190Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4542307Z   warnings.warn(
2022-11-23T02:21:02.4542517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4542851Z STAGE:2022-11-23 01:54:34 29019:29019 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4543583Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4543696Z   warnings.warn(
2022-11-23T02:21:02.4544239Z STAGE:2022-11-23 01:54:34 29020:29020 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:54:34 29019:29019 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4544260Z 
2022-11-23T02:21:02.4544829Z STAGE:2022-11-23 01:54:34 29020:29020 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:54:34 29019:29019 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4544849Z 
2022-11-23T02:21:02.4545178Z STAGE:2022-11-23 01:54:34 29020:29020 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4545504Z STAGE:2022-11-23 01:54:34 29019:29019 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4545837Z STAGE:2022-11-23 01:54:34 29020:29020 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4546166Z STAGE:2022-11-23 01:54:34 29019:29019 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4546512Z STAGE:2022-11-23 01:54:34 29020:29020 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4547067Z STAGE:2022-11-23 01:54:34 29019:29019 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4547500Z STAGE:2022-11-23 01:54:34 29020:29020 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4547838Z STAGE:2022-11-23 01:54:34 29019:29019 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4548172Z STAGE:2022-11-23 01:54:34 29020:29020 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4548495Z STAGE:2022-11-23 01:54:34 29019:29019 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4548838Z STAGE:2022-11-23 01:54:34 29020:29020 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4549184Z STAGE:2022-11-23 01:54:34 29019:29019 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4549284Z ok (4.331s)
2022-11-23T02:21:02.4549305Z 
2022-11-23T02:21:02.4549569Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4549663Z Ran 1 test in 4.331s
2022-11-23T02:21:02.4549682Z 
2022-11-23T02:21:02.4549775Z OK
2022-11-23T02:21:02.4549794Z 
2022-11-23T02:21:02.4549917Z Generating XML reports...
2022-11-23T02:21:02.4550369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015430.xml
2022-11-23T02:21:02.4550738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4550912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4551294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4551562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4551582Z 
2022-11-23T02:21:02.4551671Z Running tests...
2022-11-23T02:21:02.4551934Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4552252Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4552533Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4552751Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29132
2022-11-23T02:21:02.4552970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29133
2022-11-23T02:21:02.4553341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4553514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4553897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4554069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4554434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4554605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4554982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4555169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4555416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4555661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4556066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4556446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4556733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4556969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4557208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.4557486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.4557890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4558279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4558617Z STAGE:2022-11-23 01:54:41 29133:29133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4558940Z STAGE:2022-11-23 01:54:41 29132:29132 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4559682Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4559777Z   warnings.warn(
2022-11-23T02:21:02.4560511Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4560680Z   warnings.warn(
2022-11-23T02:21:02.4561024Z STAGE:2022-11-23 01:54:41 29132:29132 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4561352Z STAGE:2022-11-23 01:54:41 29133:29133 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4561706Z STAGE:2022-11-23 01:54:41 29132:29132 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4562053Z STAGE:2022-11-23 01:54:41 29133:29133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4562377Z STAGE:2022-11-23 01:54:41 29132:29132 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4562696Z STAGE:2022-11-23 01:54:41 29133:29133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4563008Z STAGE:2022-11-23 01:54:41 29132:29132 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4563341Z STAGE:2022-11-23 01:54:41 29133:29133 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4563685Z STAGE:2022-11-23 01:54:41 29132:29132 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4564033Z STAGE:2022-11-23 01:54:41 29133:29133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4564362Z STAGE:2022-11-23 01:54:41 29132:29132 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4564683Z STAGE:2022-11-23 01:54:41 29133:29133 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4565017Z STAGE:2022-11-23 01:54:41 29132:29132 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4565342Z STAGE:2022-11-23 01:54:41 29133:29133 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4565684Z STAGE:2022-11-23 01:54:41 29132:29132 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4566006Z STAGE:2022-11-23 01:54:41 29133:29133 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4566110Z ok (4.232s)
2022-11-23T02:21:02.4566131Z 
2022-11-23T02:21:02.4566393Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4566507Z Ran 1 test in 4.233s
2022-11-23T02:21:02.4566527Z 
2022-11-23T02:21:02.4566617Z OK
2022-11-23T02:21:02.4566686Z 
2022-11-23T02:21:02.4566815Z Generating XML reports...
2022-11-23T02:21:02.4567271Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015437.xml
2022-11-23T02:21:02.4567642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4567819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4568180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4568378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4568398Z 
2022-11-23T02:21:02.4568503Z Running tests...
2022-11-23T02:21:02.4568765Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4569076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4569352Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4569571Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29251
2022-11-23T02:21:02.4569789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29252
2022-11-23T02:21:02.4570138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4570314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4570689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4570935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4571299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4571475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4571848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4572038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4572282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4572511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4572911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4573306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4573535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4573767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4573925Z skip: Skipped due to small world size. (4.243s)
2022-11-23T02:21:02.4573945Z 
2022-11-23T02:21:02.4574212Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4574321Z Ran 1 test in 4.243s
2022-11-23T02:21:02.4574341Z 
2022-11-23T02:21:02.4574428Z OK (skipped=1)
2022-11-23T02:21:02.4574464Z 
2022-11-23T02:21:02.4574568Z Generating XML reports...
2022-11-23T02:21:02.4575014Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015444.xml
2022-11-23T02:21:02.4575388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4575562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4575988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4576186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4576206Z 
2022-11-23T02:21:02.4576313Z Running tests...
2022-11-23T02:21:02.4576577Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4576872Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4577146Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4577363Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29360
2022-11-23T02:21:02.4577586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29361
2022-11-23T02:21:02.4577959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4578135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4578516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4578705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4579069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4579224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4579593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4579888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4580133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4580379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4580790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4581184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4581417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4581732Z STAGE:2022-11-23 01:54:55 29361:29361 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4582468Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4582586Z   warnings.warn(
2022-11-23T02:21:02.4582813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4583146Z STAGE:2022-11-23 01:54:55 29360:29360 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4583877Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4583986Z   warnings.warn(
2022-11-23T02:21:02.4584529Z STAGE:2022-11-23 01:54:55 29361:29361 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:54:55 29360:29360 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4584554Z 
2022-11-23T02:21:02.4585117Z STAGE:2022-11-23 01:54:55 29360:29360 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:54:55 29361:29361 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4585137Z 
2022-11-23T02:21:02.4585705Z STAGE:2022-11-23 01:54:55 29360:29360 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:54:55 29361:29361 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4585727Z 
2022-11-23T02:21:02.4586069Z STAGE:2022-11-23 01:54:55 29360:29360 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4586396Z STAGE:2022-11-23 01:54:55 29361:29361 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4586742Z STAGE:2022-11-23 01:54:55 29360:29360 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4587341Z STAGE:2022-11-23 01:54:55 29361:29361 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4587672Z STAGE:2022-11-23 01:54:55 29360:29360 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4587999Z STAGE:2022-11-23 01:54:55 29361:29361 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4588339Z STAGE:2022-11-23 01:54:55 29360:29360 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4588666Z STAGE:2022-11-23 01:54:55 29361:29361 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4589007Z STAGE:2022-11-23 01:54:55 29360:29360 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4589347Z STAGE:2022-11-23 01:54:55 29361:29361 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4589448Z ok (4.359s)
2022-11-23T02:21:02.4589469Z 
2022-11-23T02:21:02.4589815Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4589929Z Ran 1 test in 4.359s
2022-11-23T02:21:02.4589949Z 
2022-11-23T02:21:02.4590040Z OK
2022-11-23T02:21:02.4590059Z 
2022-11-23T02:21:02.4590181Z Generating XML reports...
2022-11-23T02:21:02.4590638Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015451.xml
2022-11-23T02:21:02.4591010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4591185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4591565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4591757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4591777Z 
2022-11-23T02:21:02.4591867Z Running tests...
2022-11-23T02:21:02.4592126Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4592442Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4592722Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4592947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29473
2022-11-23T02:21:02.4593165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29474
2022-11-23T02:21:02.4593532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4593706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4594065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4594255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4594621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4594791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4595162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4595417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4595673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4595918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4596322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4596698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4596936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4597269Z STAGE:2022-11-23 01:55:02 29474:29474 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4598004Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4598116Z   warnings.warn(
2022-11-23T02:21:02.4598346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4598678Z STAGE:2022-11-23 01:55:02 29473:29473 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4599406Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2510: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4599572Z   warnings.warn(
2022-11-23T02:21:02.4599916Z STAGE:2022-11-23 01:55:02 29473:29473 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4600230Z STAGE:2022-11-23 01:55:02 29474:29474 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4600580Z STAGE:2022-11-23 01:55:02 29473:29473 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4600923Z STAGE:2022-11-23 01:55:02 29474:29474 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4601025Z ok (4.354s)
2022-11-23T02:21:02.4601044Z 
2022-11-23T02:21:02.4601307Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4601416Z Ran 1 test in 4.355s
2022-11-23T02:21:02.4601439Z 
2022-11-23T02:21:02.4601529Z OK
2022-11-23T02:21:02.4601547Z 
2022-11-23T02:21:02.4601671Z Generating XML reports...
2022-11-23T02:21:02.4602105Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015458.xml
2022-11-23T02:21:02.4602476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4602651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4603032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4603223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4603243Z 
2022-11-23T02:21:02.4603349Z Running tests...
2022-11-23T02:21:02.4603607Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4603918Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4604180Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4604381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29586
2022-11-23T02:21:02.4604600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29587
2022-11-23T02:21:02.4605021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4605203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4605583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4605773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4606134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4606314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4606666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4606856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4607103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4607345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4607744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4608138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4608366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4608779Z STAGE:2022-11-23 01:55:08 29587:29587 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4609006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4609318Z STAGE:2022-11-23 01:55:08 29586:29586 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4609656Z STAGE:2022-11-23 01:55:08 29586:29586 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4609989Z STAGE:2022-11-23 01:55:08 29587:29587 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4610332Z STAGE:2022-11-23 01:55:08 29586:29586 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4610675Z STAGE:2022-11-23 01:55:08 29587:29587 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4611002Z STAGE:2022-11-23 01:55:08 29586:29586 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4611327Z STAGE:2022-11-23 01:55:08 29587:29587 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4611657Z STAGE:2022-11-23 01:55:08 29586:29586 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4611982Z STAGE:2022-11-23 01:55:08 29587:29587 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4612314Z STAGE:2022-11-23 01:55:08 29586:29586 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4612661Z STAGE:2022-11-23 01:55:08 29587:29587 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4612760Z ok (4.359s)
2022-11-23T02:21:02.4612781Z 
2022-11-23T02:21:02.4613046Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4613156Z Ran 1 test in 4.359s
2022-11-23T02:21:02.4613176Z 
2022-11-23T02:21:02.4613269Z OK
2022-11-23T02:21:02.4613287Z 
2022-11-23T02:21:02.4613412Z Generating XML reports...
2022-11-23T02:21:02.4613868Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015505.xml
2022-11-23T02:21:02.4614235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4614446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4614841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4615031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4615051Z 
2022-11-23T02:21:02.4615157Z Running tests...
2022-11-23T02:21:02.4615416Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4615727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4615993Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-11-23T02:21:02.4616017Z 
2022-11-23T02:21:02.4616269Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4616360Z Ran 1 test in 0.002s
2022-11-23T02:21:02.4616397Z 
2022-11-23T02:21:02.4616486Z OK (skipped=1)
2022-11-23T02:21:02.4616505Z 
2022-11-23T02:21:02.4616630Z Generating XML reports...
2022-11-23T02:21:02.4617077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015511.xml
2022-11-23T02:21:02.4617444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4617618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4617996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4618187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4618256Z 
2022-11-23T02:21:02.4618368Z Running tests...
2022-11-23T02:21:02.4618615Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4618925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4619199Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-11-23T02:21:02.4619219Z 
2022-11-23T02:21:02.4619475Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4619583Z Ran 1 test in 0.002s
2022-11-23T02:21:02.4619603Z 
2022-11-23T02:21:02.4619707Z OK (skipped=1)
2022-11-23T02:21:02.4619726Z 
2022-11-23T02:21:02.4619845Z Generating XML reports...
2022-11-23T02:21:02.4620291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015514.xml
2022-11-23T02:21:02.4620657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4620817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4621191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4621382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4621402Z 
2022-11-23T02:21:02.4621508Z Running tests...
2022-11-23T02:21:02.4621766Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4622074Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4622336Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4622553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29765
2022-11-23T02:21:02.4622753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29766
2022-11-23T02:21:02.4623131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4623305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4623728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4623926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4624293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4624463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4624830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4625018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4625251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4625494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4625899Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4626293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4626521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4626749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4627212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.4627455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.4627947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4628320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4628661Z STAGE:2022-11-23 01:55:20 29765:29765 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4628982Z STAGE:2022-11-23 01:55:20 29766:29766 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4629522Z STAGE:2022-11-23 01:55:20 29766:29766 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:55:20 29765:29765 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4629544Z 
2022-11-23T02:21:02.4629893Z STAGE:2022-11-23 01:55:20 29766:29766 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4630238Z STAGE:2022-11-23 01:55:20 29765:29765 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4630560Z STAGE:2022-11-23 01:55:20 29766:29766 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4630879Z STAGE:2022-11-23 01:55:20 29765:29765 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4631421Z STAGE:2022-11-23 01:55:20 29765:29765 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:55:20 29766:29766 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4631441Z 
2022-11-23T02:21:02.4632009Z STAGE:2022-11-23 01:55:20 29766:29766 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:55:20 29765:29765 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4632029Z 
2022-11-23T02:21:02.4632128Z ok (4.238s)
2022-11-23T02:21:02.4632147Z 
2022-11-23T02:21:02.4632391Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4632506Z Ran 1 test in 4.239s
2022-11-23T02:21:02.4632525Z 
2022-11-23T02:21:02.4632616Z OK
2022-11-23T02:21:02.4632634Z 
2022-11-23T02:21:02.4632755Z Generating XML reports...
2022-11-23T02:21:02.4633266Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015516.xml
2022-11-23T02:21:02.4633651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4633825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4634202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4634392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4634411Z 
2022-11-23T02:21:02.4634501Z Running tests...
2022-11-23T02:21:02.4634762Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4635081Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4635336Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4635560Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29884
2022-11-23T02:21:02.4635777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29885
2022-11-23T02:21:02.4636147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4636323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4636678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4636867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4637300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4637476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4637845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4638034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4638281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4638524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4638922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4639302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4639534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4639762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4639920Z skip: Skipped due to small world size. (4.259s)
2022-11-23T02:21:02.4639939Z 
2022-11-23T02:21:02.4640205Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4640314Z Ran 1 test in 4.260s
2022-11-23T02:21:02.4640334Z 
2022-11-23T02:21:02.4640439Z OK (skipped=1)
2022-11-23T02:21:02.4640458Z 
2022-11-23T02:21:02.4640580Z Generating XML reports...
2022-11-23T02:21:02.4641030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015523.xml
2022-11-23T02:21:02.4641384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4641560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4641940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4642132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4642152Z 
2022-11-23T02:21:02.4642257Z Running tests...
2022-11-23T02:21:02.4642569Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4642890Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4643191Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-11-23T02:21:02.4643211Z 
2022-11-23T02:21:02.4643466Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4643561Z Ran 1 test in 0.002s
2022-11-23T02:21:02.4643579Z 
2022-11-23T02:21:02.4643684Z OK (skipped=1)
2022-11-23T02:21:02.4643706Z 
2022-11-23T02:21:02.4643826Z Generating XML reports...
2022-11-23T02:21:02.4644273Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015530.xml
2022-11-23T02:21:02.4644641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4644822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4645204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4645397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4645416Z 
2022-11-23T02:21:02.4645506Z Running tests...
2022-11-23T02:21:02.4645765Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4646074Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4646428Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-11-23T02:21:02.4646448Z 
2022-11-23T02:21:02.4646713Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4646827Z Ran 1 test in 0.002s
2022-11-23T02:21:02.4646846Z 
2022-11-23T02:21:02.4646955Z OK (skipped=1)
2022-11-23T02:21:02.4646975Z 
2022-11-23T02:21:02.4647097Z Generating XML reports...
2022-11-23T02:21:02.4647545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015532.xml
2022-11-23T02:21:02.4647894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4648071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4648451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4648642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4648661Z 
2022-11-23T02:21:02.4648768Z Running tests...
2022-11-23T02:21:02.4649029Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4649341Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4649626Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-11-23T02:21:02.4649645Z 
2022-11-23T02:21:02.4649904Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4649997Z Ran 1 test in 0.002s
2022-11-23T02:21:02.4650015Z 
2022-11-23T02:21:02.4650122Z OK (skipped=1)
2022-11-23T02:21:02.4650141Z 
2022-11-23T02:21:02.4650264Z Generating XML reports...
2022-11-23T02:21:02.4650708Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015535.xml
2022-11-23T02:21:02.4651082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4651257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4651696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4651894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4651914Z 
2022-11-23T02:21:02.4652028Z Running tests...
2022-11-23T02:21:02.4652273Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4652582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4652879Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-11-23T02:21:02.4652902Z 
2022-11-23T02:21:02.4653161Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4653270Z Ran 1 test in 0.002s
2022-11-23T02:21:02.4653289Z 
2022-11-23T02:21:02.4653395Z OK (skipped=1)
2022-11-23T02:21:02.4653414Z 
2022-11-23T02:21:02.4653533Z Generating XML reports...
2022-11-23T02:21:02.4653980Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015537.xml
2022-11-23T02:21:02.4654333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4654510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4654886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4655073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4655092Z 
2022-11-23T02:21:02.4655253Z Running tests...
2022-11-23T02:21:02.4655517Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4655830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4656105Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4656327Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30125
2022-11-23T02:21:02.4656527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30126
2022-11-23T02:21:02.4656898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4657072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4657487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4657677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4658049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4658221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4658596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4658766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4659015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4659262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4659666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4660058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4660289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4660519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4660618Z ok (4.130s)
2022-11-23T02:21:02.4660638Z 
2022-11-23T02:21:02.4660948Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4661046Z Ran 1 test in 4.130s
2022-11-23T02:21:02.4661065Z 
2022-11-23T02:21:02.4661161Z OK
2022-11-23T02:21:02.4661179Z 
2022-11-23T02:21:02.4661304Z Generating XML reports...
2022-11-23T02:21:02.4661753Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015539.xml
2022-11-23T02:21:02.4662120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4662298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4662676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4662867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4662886Z 
2022-11-23T02:21:02.4662992Z Running tests...
2022-11-23T02:21:02.4663236Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4663547Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4663824Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4664042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30234
2022-11-23T02:21:02.4664256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30235
2022-11-23T02:21:02.4664624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4664852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4665233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4665410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4665769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4665940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4666309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4666497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4666742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4667219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4667627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4668025Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4668237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4668465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4668702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.4668944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.4669342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4669739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4669979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:21:02.4670303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:21:02.4670707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.4671082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.4671323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-11-23T02:21:02.4671560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-11-23T02:21:02.4671952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:21:02.4672343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-11-23T02:21:02.4672443Z ok (4.441s)
2022-11-23T02:21:02.4672468Z 
2022-11-23T02:21:02.4672732Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4672845Z Ran 1 test in 4.441s
2022-11-23T02:21:02.4672865Z 
2022-11-23T02:21:02.4672958Z OK
2022-11-23T02:21:02.4672978Z 
2022-11-23T02:21:02.4673082Z Generating XML reports...
2022-11-23T02:21:02.4673535Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015546.xml
2022-11-23T02:21:02.4673902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4674165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4674548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4674738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4674757Z 
2022-11-23T02:21:02.4674868Z Running tests...
2022-11-23T02:21:02.4675130Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4675424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4675683Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-11-23T02:21:02.4675703Z 
2022-11-23T02:21:02.4675957Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4676069Z Ran 1 test in 0.003s
2022-11-23T02:21:02.4676087Z 
2022-11-23T02:21:02.4676194Z OK (skipped=1)
2022-11-23T02:21:02.4676217Z 
2022-11-23T02:21:02.4676338Z Generating XML reports...
2022-11-23T02:21:02.4676784Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015553.xml
2022-11-23T02:21:02.4677152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4677333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4677697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4677887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4677906Z 
2022-11-23T02:21:02.4678012Z Running tests...
2022-11-23T02:21:02.4678271Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4678581Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4678868Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4679086Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30400
2022-11-23T02:21:02.4679301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30401
2022-11-23T02:21:02.4679723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4679886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4680267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4680454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4680815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4680994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4681366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4681555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4681801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4682025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4682420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4682812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4683044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4683327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4683567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.4683807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.4684208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4684598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4684931Z STAGE:2022-11-23 01:55:59 30400:30400 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4685653Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4685772Z   warnings.warn(
2022-11-23T02:21:02.4686105Z STAGE:2022-11-23 01:55:59 30401:30401 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4686840Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4686952Z   warnings.warn(
2022-11-23T02:21:02.4687292Z STAGE:2022-11-23 01:55:59 30400:30400 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4687623Z STAGE:2022-11-23 01:55:59 30401:30401 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4687971Z STAGE:2022-11-23 01:55:59 30400:30400 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4688318Z STAGE:2022-11-23 01:55:59 30401:30401 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4688843Z STAGE:2022-11-23 01:55:59 30401:30401 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:55:59 30400:30400 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4688864Z 
2022-11-23T02:21:02.4689231Z STAGE:2022-11-23 01:55:59 30400:30400 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4689574Z STAGE:2022-11-23 01:55:59 30401:30401 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4690138Z STAGE:2022-11-23 01:55:59 30400:30400 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:55:59 30401:30401 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4690158Z 
2022-11-23T02:21:02.4690260Z ok (4.357s)
2022-11-23T02:21:02.4690280Z 
2022-11-23T02:21:02.4690548Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4690657Z Ran 1 test in 4.358s
2022-11-23T02:21:02.4690676Z 
2022-11-23T02:21:02.4690765Z OK
2022-11-23T02:21:02.4690784Z 
2022-11-23T02:21:02.4690906Z Generating XML reports...
2022-11-23T02:21:02.4691360Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015555.xml
2022-11-23T02:21:02.4691729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4691889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4692263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4692451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4692470Z 
2022-11-23T02:21:02.4692578Z Running tests...
2022-11-23T02:21:02.4692900Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4693211Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4693493Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4693718Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30519
2022-11-23T02:21:02.4693918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30520
2022-11-23T02:21:02.4694287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4694459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4694836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4695022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4695384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4695556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4695929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4696114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4696344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4696583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4696981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4697372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4697605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4697832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4698117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.4698365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.4698764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4699140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4699474Z STAGE:2022-11-23 01:56:06 30519:30519 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4700205Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4700320Z   warnings.warn(
2022-11-23T02:21:02.4700653Z STAGE:2022-11-23 01:56:06 30520:30520 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4701381Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4701489Z   warnings.warn(
2022-11-23T02:21:02.4701824Z STAGE:2022-11-23 01:56:06 30519:30519 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4702147Z STAGE:2022-11-23 01:56:06 30520:30520 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4702777Z STAGE:2022-11-23 01:56:06 30519:30519 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:56:06 30520:30520 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4702799Z 
2022-11-23T02:21:02.4703111Z STAGE:2022-11-23 01:56:06 30519:30519 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4703424Z STAGE:2022-11-23 01:56:06 30520:30520 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4703749Z STAGE:2022-11-23 01:56:06 30520:30520 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4704080Z STAGE:2022-11-23 01:56:06 30519:30519 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4704421Z STAGE:2022-11-23 01:56:06 30520:30520 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4704765Z STAGE:2022-11-23 01:56:06 30519:30519 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4704870Z ok (4.343s)
2022-11-23T02:21:02.4704889Z 
2022-11-23T02:21:02.4705150Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4705258Z Ran 1 test in 4.344s
2022-11-23T02:21:02.4705277Z 
2022-11-23T02:21:02.4705349Z OK
2022-11-23T02:21:02.4705368Z 
2022-11-23T02:21:02.4705491Z Generating XML reports...
2022-11-23T02:21:02.4705938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015602.xml
2022-11-23T02:21:02.4706306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4706480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4707155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4707470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4707499Z 
2022-11-23T02:21:02.4707685Z Running tests...
2022-11-23T02:21:02.4707967Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4708262Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4708648Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4708875Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30638
2022-11-23T02:21:02.4709084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30639
2022-11-23T02:21:02.4709449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4709614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4709981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4710165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4710513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4710679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4711040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4711215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4711451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4711683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4712072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4712536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4712759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4712974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4713202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.4713432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.4713819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4714199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4714526Z STAGE:2022-11-23 01:56:13 30638:30638 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4714838Z STAGE:2022-11-23 01:56:13 30639:30639 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4715570Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4715672Z   warnings.warn(
2022-11-23T02:21:02.4716396Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4716488Z   warnings.warn(
2022-11-23T02:21:02.4716824Z STAGE:2022-11-23 01:56:13 30638:30638 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4717140Z STAGE:2022-11-23 01:56:13 30639:30639 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4717749Z STAGE:2022-11-23 01:56:13 30638:30638 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:56:13 30639:30639 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4717771Z 
2022-11-23T02:21:02.4718100Z STAGE:2022-11-23 01:56:13 30639:30639 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4718410Z STAGE:2022-11-23 01:56:13 30638:30638 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4718871Z STAGE:2022-11-23 01:56:13 30639:30639 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4719337Z STAGE:2022-11-23 01:56:13 30638:30638 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4719685Z STAGE:2022-11-23 01:56:13 30639:30639 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4720020Z STAGE:2022-11-23 01:56:13 30638:30638 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4720104Z ok (4.374s)
2022-11-23T02:21:02.4720123Z 
2022-11-23T02:21:02.4720380Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4720483Z Ran 1 test in 4.374s
2022-11-23T02:21:02.4720503Z 
2022-11-23T02:21:02.4720585Z OK
2022-11-23T02:21:02.4720603Z 
2022-11-23T02:21:02.4720716Z Generating XML reports...
2022-11-23T02:21:02.4721158Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015609.xml
2022-11-23T02:21:02.4721519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4721685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4722128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4722309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4722329Z 
2022-11-23T02:21:02.4722428Z Running tests...
2022-11-23T02:21:02.4722685Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4722987Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4723264Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4723473Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30757
2022-11-23T02:21:02.4723678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30758
2022-11-23T02:21:02.4724040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4724202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4724575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4724752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4725107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4725270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4725630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4725806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4726040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4726269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4726659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4727141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4727370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4727587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4727816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.4728048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.4728437Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4728825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4729141Z STAGE:2022-11-23 01:56:20 30757:30757 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4729871Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4729973Z   warnings.warn(
2022-11-23T02:21:02.4730294Z STAGE:2022-11-23 01:56:20 30758:30758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4731015Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4731170Z   warnings.warn(
2022-11-23T02:21:02.4731501Z STAGE:2022-11-23 01:56:20 30758:30758 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4731823Z STAGE:2022-11-23 01:56:20 30757:30757 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4732160Z STAGE:2022-11-23 01:56:20 30758:30758 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4732494Z STAGE:2022-11-23 01:56:20 30757:30757 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4732801Z STAGE:2022-11-23 01:56:20 30758:30758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4733109Z STAGE:2022-11-23 01:56:20 30757:30757 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4733432Z STAGE:2022-11-23 01:56:20 30758:30758 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4733758Z STAGE:2022-11-23 01:56:20 30757:30757 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4734091Z STAGE:2022-11-23 01:56:20 30758:30758 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4734426Z STAGE:2022-11-23 01:56:20 30757:30757 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4734519Z ok (4.369s)
2022-11-23T02:21:02.4734540Z 
2022-11-23T02:21:02.4734794Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4734894Z Ran 1 test in 4.369s
2022-11-23T02:21:02.4734913Z 
2022-11-23T02:21:02.4734986Z OK
2022-11-23T02:21:02.4735004Z 
2022-11-23T02:21:02.4735117Z Generating XML reports...
2022-11-23T02:21:02.4735564Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015616.xml
2022-11-23T02:21:02.4735925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4736097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4736472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4736703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4736725Z 
2022-11-23T02:21:02.4736827Z Running tests...
2022-11-23T02:21:02.4737076Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4737376Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4737647Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4737856Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30876
2022-11-23T02:21:02.4738065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30877
2022-11-23T02:21:02.4738427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4738591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4738961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4739141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4739490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4739653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4740021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4740212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4740536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4740782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4741189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4741575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4741797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4742006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4742158Z skip: Skipped due to small world size. (4.242s)
2022-11-23T02:21:02.4742177Z 
2022-11-23T02:21:02.4742439Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4742553Z Ran 1 test in 4.242s
2022-11-23T02:21:02.4742573Z 
2022-11-23T02:21:02.4742677Z OK (skipped=1)
2022-11-23T02:21:02.4742695Z 
2022-11-23T02:21:02.4742817Z Generating XML reports...
2022-11-23T02:21:02.4743318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015623.xml
2022-11-23T02:21:02.4743702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4743861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4744232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4744415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4744434Z 
2022-11-23T02:21:02.4744537Z Running tests...
2022-11-23T02:21:02.4744791Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4745102Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4745373Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4745587Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30985
2022-11-23T02:21:02.4745853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30986
2022-11-23T02:21:02.4746217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4746388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4746933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4747142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4747519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4747692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4748071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4748259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4748486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4748722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4749118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4749501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4749822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4750041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4750189Z skip: Skipped due to small world size. (4.119s)
2022-11-23T02:21:02.4750210Z 
2022-11-23T02:21:02.4750471Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4750573Z Ran 1 test in 4.119s
2022-11-23T02:21:02.4750593Z 
2022-11-23T02:21:02.4750680Z OK (skipped=1)
2022-11-23T02:21:02.4750699Z 
2022-11-23T02:21:02.4750813Z Generating XML reports...
2022-11-23T02:21:02.4751257Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015630.xml
2022-11-23T02:21:02.4751621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4751785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4752157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4752337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4752357Z 
2022-11-23T02:21:02.4752454Z Running tests...
2022-11-23T02:21:02.4752709Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4753005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4753283Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4753491Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31094
2022-11-23T02:21:02.4753697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31095
2022-11-23T02:21:02.4754059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4754228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4754597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4754776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4755186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4755358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4755722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4755897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4756131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4756368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4756757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4757144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4757370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4757619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4757773Z skip: Skipped due to small world size. (4.254s)
2022-11-23T02:21:02.4757792Z 
2022-11-23T02:21:02.4758058Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4758158Z Ran 1 test in 4.254s
2022-11-23T02:21:02.4758177Z 
2022-11-23T02:21:02.4758279Z OK (skipped=1)
2022-11-23T02:21:02.4758351Z 
2022-11-23T02:21:02.4758476Z Generating XML reports...
2022-11-23T02:21:02.4758924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015636.xml
2022-11-23T02:21:02.4759288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4759461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4759821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4760003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4760023Z 
2022-11-23T02:21:02.4760119Z Running tests...
2022-11-23T02:21:02.4760370Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4760671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4760943Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4761153Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31203
2022-11-23T02:21:02.4761366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31204
2022-11-23T02:21:02.4761723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4761888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4762257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4762436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4762787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4762949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4763316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4763494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4763778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4764010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4764400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4764783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4765003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4765220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4765373Z skip: Skipped due to small world size. (4.160s)
2022-11-23T02:21:02.4765392Z 
2022-11-23T02:21:02.4765645Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4765748Z Ran 1 test in 4.161s
2022-11-23T02:21:02.4765767Z 
2022-11-23T02:21:02.4765865Z OK (skipped=1)
2022-11-23T02:21:02.4765888Z 
2022-11-23T02:21:02.4765993Z Generating XML reports...
2022-11-23T02:21:02.4766435Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015643.xml
2022-11-23T02:21:02.4766792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4766956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4767323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4767557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4767577Z 
2022-11-23T02:21:02.4767674Z Running tests...
2022-11-23T02:21:02.4767931Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4768227Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4768486Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4768693Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31312
2022-11-23T02:21:02.4768899Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31313
2022-11-23T02:21:02.4769255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4769419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4769793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4769972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4770321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4770479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4770846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4771022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4771256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4771489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4771874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4772263Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4772482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4772742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4773082Z STAGE:2022-11-23 01:56:54 31313:31313 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4773395Z STAGE:2022-11-23 01:56:54 31312:31312 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4774130Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4774244Z   warnings.warn(
2022-11-23T02:21:02.4774970Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4775077Z   warnings.warn(
2022-11-23T02:21:02.4775403Z STAGE:2022-11-23 01:56:54 31313:31313 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4775730Z STAGE:2022-11-23 01:56:54 31312:31312 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4776068Z STAGE:2022-11-23 01:56:54 31313:31313 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4776395Z STAGE:2022-11-23 01:56:54 31312:31312 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4776719Z STAGE:2022-11-23 01:56:54 31313:31313 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4777091Z STAGE:2022-11-23 01:56:54 31312:31312 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4777423Z STAGE:2022-11-23 01:56:54 31313:31313 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4777753Z STAGE:2022-11-23 01:56:54 31312:31312 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4778091Z STAGE:2022-11-23 01:56:54 31313:31313 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4778429Z STAGE:2022-11-23 01:56:54 31312:31312 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4778522Z ok (4.143s)
2022-11-23T02:21:02.4778542Z 
2022-11-23T02:21:02.4778804Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4778897Z Ran 1 test in 4.144s
2022-11-23T02:21:02.4778916Z 
2022-11-23T02:21:02.4779013Z OK
2022-11-23T02:21:02.4779033Z 
2022-11-23T02:21:02.4779151Z Generating XML reports...
2022-11-23T02:21:02.4779595Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015650.xml
2022-11-23T02:21:02.4779966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4780143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4780516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4780710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4780730Z 
2022-11-23T02:21:02.4780837Z Running tests...
2022-11-23T02:21:02.4781084Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4781394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4781700Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4781909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31425
2022-11-23T02:21:02.4782114Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31426
2022-11-23T02:21:02.4782527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4782701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4783075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4783245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4783596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4783764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4784124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4784300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4784538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4784774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4785162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4785554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4785766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4786551Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4786661Z   warnings.warn(
2022-11-23T02:21:02.4787228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4787978Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4788082Z   warnings.warn(
2022-11-23T02:21:02.4788179Z ok (4.377s)
2022-11-23T02:21:02.4788199Z 
2022-11-23T02:21:02.4788461Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4788576Z Ran 1 test in 4.377s
2022-11-23T02:21:02.4788596Z 
2022-11-23T02:21:02.4788669Z OK
2022-11-23T02:21:02.4788698Z 
2022-11-23T02:21:02.4788803Z Generating XML reports...
2022-11-23T02:21:02.4789254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015657.xml
2022-11-23T02:21:02.4789616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4789788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4790159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4790348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4790368Z 
2022-11-23T02:21:02.4790466Z Running tests...
2022-11-23T02:21:02.4790724Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4791024Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4791288Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4791507Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31534
2022-11-23T02:21:02.4791802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31535
2022-11-23T02:21:02.4792182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4792348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4792719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4792901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4793243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4793417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4793779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4793966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4794210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4794451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4794840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4795231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4795452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4795843Z STAGE:2022-11-23 01:57:07 31535:31535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4796577Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4796681Z   warnings.warn(
2022-11-23T02:21:02.4796905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4797230Z STAGE:2022-11-23 01:57:08 31534:31534 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4797956Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4798064Z   warnings.warn(
2022-11-23T02:21:02.4798402Z STAGE:2022-11-23 01:57:08 31534:31534 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4798722Z STAGE:2022-11-23 01:57:08 31535:31535 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4799290Z STAGE:2022-11-23 01:57:08 31535:31535 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:57:08 31534:31534 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4799311Z 
2022-11-23T02:21:02.4799626Z STAGE:2022-11-23 01:57:08 31534:31534 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4799927Z STAGE:2022-11-23 01:57:08 31535:31535 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4800254Z STAGE:2022-11-23 01:57:08 31535:31535 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4800583Z STAGE:2022-11-23 01:57:08 31534:31534 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4800924Z STAGE:2022-11-23 01:57:08 31535:31535 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4801310Z STAGE:2022-11-23 01:57:08 31534:31534 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4801413Z ok (4.361s)
2022-11-23T02:21:02.4801433Z 
2022-11-23T02:21:02.4801691Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4801803Z Ran 1 test in 4.361s
2022-11-23T02:21:02.4801822Z 
2022-11-23T02:21:02.4801894Z OK
2022-11-23T02:21:02.4801931Z 
2022-11-23T02:21:02.4802036Z Generating XML reports...
2022-11-23T02:21:02.4802478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015704.xml
2022-11-23T02:21:02.4802847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4803021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4803402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4803586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4803606Z 
2022-11-23T02:21:02.4803711Z Running tests...
2022-11-23T02:21:02.4803965Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4804258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4804533Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4804740Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31647
2022-11-23T02:21:02.4805029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31648
2022-11-23T02:21:02.4805395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4805571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4805951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4806136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4806481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4806651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4807011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4807195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4807436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4807676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4808072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4808466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4808695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4809011Z STAGE:2022-11-23 01:57:14 31648:31648 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4809739Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4809852Z   warnings.warn(
2022-11-23T02:21:02.4810071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4810451Z STAGE:2022-11-23 01:57:14 31647:31647 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4811188Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4811298Z   warnings.warn(
2022-11-23T02:21:02.4811630Z STAGE:2022-11-23 01:57:14 31647:31647 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4811957Z STAGE:2022-11-23 01:57:14 31648:31648 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4812300Z STAGE:2022-11-23 01:57:14 31647:31647 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4812626Z STAGE:2022-11-23 01:57:14 31648:31648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4813158Z STAGE:2022-11-23 01:57:14 31648:31648 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:57:14 31647:31647 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4813179Z 
2022-11-23T02:21:02.4813716Z STAGE:2022-11-23 01:57:14 31647:31647 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:57:14 31648:31648 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4813737Z 
2022-11-23T02:21:02.4814075Z STAGE:2022-11-23 01:57:14 31648:31648 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4814416Z STAGE:2022-11-23 01:57:14 31647:31647 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4814566Z ok (4.371s)
2022-11-23T02:21:02.4814586Z 
2022-11-23T02:21:02.4814851Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4814953Z Ran 1 test in 4.372s
2022-11-23T02:21:02.4814973Z 
2022-11-23T02:21:02.4815066Z OK
2022-11-23T02:21:02.4815085Z 
2022-11-23T02:21:02.4815206Z Generating XML reports...
2022-11-23T02:21:02.4815641Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015710.xml
2022-11-23T02:21:02.4816011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4816180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4816557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4816737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4816760Z 
2022-11-23T02:21:02.4816865Z Running tests...
2022-11-23T02:21:02.4817119Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4817429Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4817695Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4817898Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31760
2022-11-23T02:21:02.4818112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31761
2022-11-23T02:21:02.4818480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4818647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4819018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4819206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4819565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4819796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4820159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4820341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4820584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4820821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4821220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4821619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4821850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4822078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4822406Z STAGE:2022-11-23 01:57:21 31760:31760 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4822709Z STAGE:2022-11-23 01:57:21 31761:31761 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4823440Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4823601Z   warnings.warn(
2022-11-23T02:21:02.4824333Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1638: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-11-23T02:21:02.4824437Z   warnings.warn(
2022-11-23T02:21:02.4824780Z STAGE:2022-11-23 01:57:21 31760:31760 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4825102Z STAGE:2022-11-23 01:57:21 31761:31761 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4825444Z STAGE:2022-11-23 01:57:21 31760:31760 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4825787Z STAGE:2022-11-23 01:57:21 31761:31761 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4826111Z STAGE:2022-11-23 01:57:21 31760:31760 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4826420Z STAGE:2022-11-23 01:57:21 31761:31761 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4827179Z STAGE:2022-11-23 01:57:21 31760:31760 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:57:21 31761:31761 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4827208Z 
2022-11-23T02:21:02.4827567Z STAGE:2022-11-23 01:57:21 31760:31760 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4827908Z STAGE:2022-11-23 01:57:21 31761:31761 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4828000Z ok (4.117s)
2022-11-23T02:21:02.4828020Z 
2022-11-23T02:21:02.4828278Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4828380Z Ran 1 test in 4.117s
2022-11-23T02:21:02.4828400Z 
2022-11-23T02:21:02.4828487Z OK
2022-11-23T02:21:02.4828506Z 
2022-11-23T02:21:02.4828627Z Generating XML reports...
2022-11-23T02:21:02.4829059Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015717.xml
2022-11-23T02:21:02.4829428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4829675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4830071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4830259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4830279Z 
2022-11-23T02:21:02.4830381Z Running tests...
2022-11-23T02:21:02.4830640Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4830947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4831230Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4831439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31873
2022-11-23T02:21:02.4831653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31874
2022-11-23T02:21:02.4832025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4832194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4832568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4832750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4833110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4833277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4833713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4833900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4834140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4834384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4834775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4835166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4835394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4835618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4835722Z ok (4.345s)
2022-11-23T02:21:02.4835742Z 
2022-11-23T02:21:02.4835989Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4836093Z Ran 1 test in 4.345s
2022-11-23T02:21:02.4836113Z 
2022-11-23T02:21:02.4836201Z OK
2022-11-23T02:21:02.4836221Z 
2022-11-23T02:21:02.4836337Z Generating XML reports...
2022-11-23T02:21:02.4836781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015724.xml
2022-11-23T02:21:02.4837146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4837321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4837694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4837877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4837901Z 
2022-11-23T02:21:02.4837991Z Running tests...
2022-11-23T02:21:02.4838245Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4838555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4838873Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4839089Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31982
2022-11-23T02:21:02.4839303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31983
2022-11-23T02:21:02.4839665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4839836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4840194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4840381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4840742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4840917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4841279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4841465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4841705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4841947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4842340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4842779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4843006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4843229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4843467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.4843705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.4844095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4844485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4844816Z STAGE:2022-11-23 01:57:35 31983:31983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4845133Z STAGE:2022-11-23 01:57:35 31982:31982 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4845661Z STAGE:2022-11-23 01:57:35 31982:31982 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:57:35 31983:31983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4845693Z 
2022-11-23T02:21:02.4846243Z STAGE:2022-11-23 01:57:35 31982:31982 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:57:35 31983:31983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4846278Z 
2022-11-23T02:21:02.4846586Z STAGE:2022-11-23 01:57:35 31982:31982 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4846896Z STAGE:2022-11-23 01:57:35 31983:31983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4847230Z STAGE:2022-11-23 01:57:35 31982:31982 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4847551Z STAGE:2022-11-23 01:57:35 31983:31983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4847888Z STAGE:2022-11-23 01:57:35 31982:31982 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4848275Z STAGE:2022-11-23 01:57:35 31983:31983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4848377Z ok (4.230s)
2022-11-23T02:21:02.4848396Z 
2022-11-23T02:21:02.4848660Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4848753Z Ran 1 test in 4.231s
2022-11-23T02:21:02.4848772Z 
2022-11-23T02:21:02.4848860Z OK
2022-11-23T02:21:02.4848879Z 
2022-11-23T02:21:02.4849001Z Generating XML reports...
2022-11-23T02:21:02.4849448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015731.xml
2022-11-23T02:21:02.4849813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4849985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4850360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4850549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4850568Z 
2022-11-23T02:21:02.4850667Z Running tests...
2022-11-23T02:21:02.4850915Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4851224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4851487Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4851703Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32101
2022-11-23T02:21:02.4851973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32102
2022-11-23T02:21:02.4852344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4852518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4852892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4853064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4853423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4853588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4853955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4854140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4854381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4854620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4855018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4855406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4855618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4855848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4856089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.4856325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.4856720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4857148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4857532Z STAGE:2022-11-23 01:57:42 32101:32101 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4857847Z STAGE:2022-11-23 01:57:42 32102:32102 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4858175Z STAGE:2022-11-23 01:57:42 32101:32101 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4858484Z STAGE:2022-11-23 01:57:42 32102:32102 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4858827Z STAGE:2022-11-23 01:57:42 32101:32101 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4859177Z STAGE:2022-11-23 01:57:42 32102:32102 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4859499Z STAGE:2022-11-23 01:57:42 32102:32102 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4859817Z STAGE:2022-11-23 01:57:42 32101:32101 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4860140Z STAGE:2022-11-23 01:57:42 32102:32102 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4860464Z STAGE:2022-11-23 01:57:42 32101:32101 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4860799Z STAGE:2022-11-23 01:57:42 32102:32102 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4861142Z STAGE:2022-11-23 01:57:42 32101:32101 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4861282Z ok (4.364s)
2022-11-23T02:21:02.4861302Z 
2022-11-23T02:21:02.4861567Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4861674Z Ran 1 test in 4.364s
2022-11-23T02:21:02.4861693Z 
2022-11-23T02:21:02.4861776Z OK
2022-11-23T02:21:02.4861795Z 
2022-11-23T02:21:02.4861918Z Generating XML reports...
2022-11-23T02:21:02.4862368Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015738.xml
2022-11-23T02:21:02.4862735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4862905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4863266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4863456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4863479Z 
2022-11-23T02:21:02.4863578Z Running tests...
2022-11-23T02:21:02.4863840Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4864145Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4864422Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4864641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32220
2022-11-23T02:21:02.4864858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32221
2022-11-23T02:21:02.4865230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4865386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4865760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4865951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4866314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4866483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4867150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4867354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4867599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4867824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4868236Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4868636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4868865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4869094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4869335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.4869575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.4869965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4870356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4870692Z STAGE:2022-11-23 01:57:48 32221:32221 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4871139Z STAGE:2022-11-23 01:57:48 32220:32220 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4871473Z STAGE:2022-11-23 01:57:48 32220:32220 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4871802Z STAGE:2022-11-23 01:57:48 32221:32221 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4872150Z STAGE:2022-11-23 01:57:48 32220:32220 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4872493Z STAGE:2022-11-23 01:57:48 32221:32221 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4872816Z STAGE:2022-11-23 01:57:48 32220:32220 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4873134Z STAGE:2022-11-23 01:57:48 32221:32221 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4873466Z STAGE:2022-11-23 01:57:48 32220:32220 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4873797Z STAGE:2022-11-23 01:57:48 32221:32221 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4874127Z STAGE:2022-11-23 01:57:48 32220:32220 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4874470Z STAGE:2022-11-23 01:57:48 32221:32221 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4874569Z ok (4.248s)
2022-11-23T02:21:02.4874589Z 
2022-11-23T02:21:02.4874853Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4874965Z Ran 1 test in 4.248s
2022-11-23T02:21:02.4874984Z 
2022-11-23T02:21:02.4875075Z OK
2022-11-23T02:21:02.4875095Z 
2022-11-23T02:21:02.4875216Z Generating XML reports...
2022-11-23T02:21:02.4875669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015745.xml
2022-11-23T02:21:02.4876020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4876199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4876575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4876817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4876839Z 
2022-11-23T02:21:02.4876952Z Running tests...
2022-11-23T02:21:02.4877216Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4877531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4877797Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4878015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32339
2022-11-23T02:21:02.4878215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32340
2022-11-23T02:21:02.4878590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4878765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4879143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4879332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4879694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4879869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4880237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4880405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4880716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4880962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4881367Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4881763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4881993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4882220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4882459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.4882697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.4883079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4883469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.4883804Z STAGE:2022-11-23 01:57:55 32340:32340 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4884128Z STAGE:2022-11-23 01:57:55 32339:32339 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4884671Z STAGE:2022-11-23 01:57:55 32340:32340 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:57:55 32339:32339 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4884691Z 
2022-11-23T02:21:02.4885260Z STAGE:2022-11-23 01:57:55 32340:32340 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:57:55 32339:32339 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4885284Z 
2022-11-23T02:21:02.4885611Z STAGE:2022-11-23 01:57:55 32340:32340 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4885929Z STAGE:2022-11-23 01:57:55 32339:32339 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4886511Z STAGE:2022-11-23 01:57:55 32339:32339 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:57:55 32340:32340 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4886532Z 
2022-11-23T02:21:02.4887104Z STAGE:2022-11-23 01:57:55 32340:32340 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:57:55 32339:32339 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4887125Z 
2022-11-23T02:21:02.4887226Z ok (4.350s)
2022-11-23T02:21:02.4887245Z 
2022-11-23T02:21:02.4887512Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4887606Z Ran 1 test in 4.350s
2022-11-23T02:21:02.4887647Z 
2022-11-23T02:21:02.4887720Z OK
2022-11-23T02:21:02.4887738Z 
2022-11-23T02:21:02.4887864Z Generating XML reports...
2022-11-23T02:21:02.4888316Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015751.xml
2022-11-23T02:21:02.4888684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4888858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4889235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4889426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4889446Z 
2022-11-23T02:21:02.4889553Z Running tests...
2022-11-23T02:21:02.4889854Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4890166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4890430Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4890656Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32458
2022-11-23T02:21:02.4890872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32459
2022-11-23T02:21:02.4891244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4891419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4891790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4891963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4892330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4892505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4892870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4893065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4893308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4893553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4893952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4894345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4894561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4894788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4894944Z skip: Skipped due to small world size. (4.238s)
2022-11-23T02:21:02.4894963Z 
2022-11-23T02:21:02.4895274Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4895390Z Ran 1 test in 4.238s
2022-11-23T02:21:02.4895410Z 
2022-11-23T02:21:02.4895515Z OK (skipped=1)
2022-11-23T02:21:02.4895534Z 
2022-11-23T02:21:02.4895656Z Generating XML reports...
2022-11-23T02:21:02.4896112Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015758.xml
2022-11-23T02:21:02.4896481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4896643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4897022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4897215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4897234Z 
2022-11-23T02:21:02.4897341Z Running tests...
2022-11-23T02:21:02.4897607Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4897917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4898177Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4898395Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32567
2022-11-23T02:21:02.4898594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32568
2022-11-23T02:21:02.4898959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4899188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4899568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4899760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4900121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4900297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4900666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4900851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4901079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4901326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4901723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4902121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4902348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4902574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4902731Z skip: Skipped due to small world size. (4.355s)
2022-11-23T02:21:02.4902750Z 
2022-11-23T02:21:02.4903018Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4903111Z Ran 1 test in 4.355s
2022-11-23T02:21:02.4903146Z 
2022-11-23T02:21:02.4903234Z OK (skipped=1)
2022-11-23T02:21:02.4903256Z 
2022-11-23T02:21:02.4903377Z Generating XML reports...
2022-11-23T02:21:02.4903828Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015805.xml
2022-11-23T02:21:02.4904201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4904429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4904819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4905011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4905031Z 
2022-11-23T02:21:02.4905139Z Running tests...
2022-11-23T02:21:02.4905384Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4905693Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4905966Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4906182Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32676
2022-11-23T02:21:02.4906397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32677
2022-11-23T02:21:02.4906986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4907180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4907564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4907753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4908099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4908360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4908735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4908921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4909169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4909411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4909807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4910200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4910411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4910642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4910796Z skip: Skipped due to small world size. (4.247s)
2022-11-23T02:21:02.4910816Z 
2022-11-23T02:21:02.4911080Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4911193Z Ran 1 test in 4.248s
2022-11-23T02:21:02.4911212Z 
2022-11-23T02:21:02.4911320Z OK (skipped=1)
2022-11-23T02:21:02.4911339Z 
2022-11-23T02:21:02.4911460Z Generating XML reports...
2022-11-23T02:21:02.4911913Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015812.xml
2022-11-23T02:21:02.4912281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4912441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4912822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4913014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4913035Z 
2022-11-23T02:21:02.4913144Z Running tests...
2022-11-23T02:21:02.4913407Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4913781Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4914056Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4914276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32785
2022-11-23T02:21:02.4914476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32786
2022-11-23T02:21:02.4914850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4915025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4915405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4915593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4915952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4916130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4916507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4916694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4916922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4917164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4917560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4918016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4918243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4918473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4918629Z skip: Skipped due to small world size. (4.246s)
2022-11-23T02:21:02.4918649Z 
2022-11-23T02:21:02.4918914Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4919026Z Ran 1 test in 4.247s
2022-11-23T02:21:02.4919046Z 
2022-11-23T02:21:02.4919133Z OK (skipped=1)
2022-11-23T02:21:02.4919152Z 
2022-11-23T02:21:02.4919272Z Generating XML reports...
2022-11-23T02:21:02.4919718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015819.xml
2022-11-23T02:21:02.4920094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4920269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4920648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4920842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4920861Z 
2022-11-23T02:21:02.4920969Z Running tests...
2022-11-23T02:21:02.4921213Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4921527Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4921778Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4921994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32894
2022-11-23T02:21:02.4922215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32895
2022-11-23T02:21:02.4922584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4922757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4923188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4923386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4923740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4923913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4924283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4924474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4924720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4924961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4925365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4925762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4925991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4926311Z STAGE:2022-11-23 01:58:29 32895:32895 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4926538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4926928Z STAGE:2022-11-23 01:58:29 32894:32894 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4927267Z STAGE:2022-11-23 01:58:29 32894:32894 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4927597Z STAGE:2022-11-23 01:58:29 32895:32895 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4927946Z STAGE:2022-11-23 01:58:29 32894:32894 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4928290Z STAGE:2022-11-23 01:58:29 32895:32895 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4928814Z STAGE:2022-11-23 01:58:29 32895:32895 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:58:29 32894:32894 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4928835Z 
2022-11-23T02:21:02.4929373Z STAGE:2022-11-23 01:58:29 32895:32895 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:58:29 32894:32894 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4929398Z 
2022-11-23T02:21:02.4929966Z STAGE:2022-11-23 01:58:29 32894:32894 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:58:29 32895:32895 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4929986Z 
2022-11-23T02:21:02.4930086Z ok (4.254s)
2022-11-23T02:21:02.4930105Z 
2022-11-23T02:21:02.4930350Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4930461Z Ran 1 test in 4.255s
2022-11-23T02:21:02.4930480Z 
2022-11-23T02:21:02.4930569Z OK
2022-11-23T02:21:02.4930588Z 
2022-11-23T02:21:02.4930712Z Generating XML reports...
2022-11-23T02:21:02.4931162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015826.xml
2022-11-23T02:21:02.4931537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4931700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4932094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4932333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4932354Z 
2022-11-23T02:21:02.4932464Z Running tests...
2022-11-23T02:21:02.4932730Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4933038Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4933290Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4933506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33007
2022-11-23T02:21:02.4933704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33008
2022-11-23T02:21:02.4934078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4934250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4934629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4934819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4935178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4935349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4935720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4935905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4936201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4936444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4936848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4937244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4937475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4937808Z STAGE:2022-11-23 01:58:36 33008:33008 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4938033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4938354Z STAGE:2022-11-23 01:58:36 33007:33007 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4938694Z STAGE:2022-11-23 01:58:36 33007:33007 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4939006Z STAGE:2022-11-23 01:58:36 33008:33008 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4939353Z STAGE:2022-11-23 01:58:36 33007:33007 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4939699Z STAGE:2022-11-23 01:58:36 33008:33008 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4940223Z STAGE:2022-11-23 01:58:36 33007:33007 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:58:36 33008:33008 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4940244Z 
2022-11-23T02:21:02.4940573Z STAGE:2022-11-23 01:58:36 33008:33008 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4940896Z STAGE:2022-11-23 01:58:36 33007:33007 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4941244Z STAGE:2022-11-23 01:58:36 33008:33008 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4941583Z STAGE:2022-11-23 01:58:36 33007:33007 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4941681Z ok (4.337s)
2022-11-23T02:21:02.4941747Z 
2022-11-23T02:21:02.4942019Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4942113Z Ran 1 test in 4.337s
2022-11-23T02:21:02.4942132Z 
2022-11-23T02:21:02.4942223Z OK
2022-11-23T02:21:02.4942242Z 
2022-11-23T02:21:02.4942363Z Generating XML reports...
2022-11-23T02:21:02.4942808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015832.xml
2022-11-23T02:21:02.4943175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4943353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4943734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4943924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4943943Z 
2022-11-23T02:21:02.4944038Z Running tests...
2022-11-23T02:21:02.4944298Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4944606Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4944868Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4945086Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33120
2022-11-23T02:21:02.4945302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33121
2022-11-23T02:21:02.4945669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4945899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4946263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4946457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4947032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4947212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4947593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4947781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4948029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4948279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4948677Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4949060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4949291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4949515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4949850Z STAGE:2022-11-23 01:58:45 33121:33121 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4950611Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:21:02.4950727Z   warnings.warn(
2022-11-23T02:21:02.4951059Z STAGE:2022-11-23 01:58:45 33120:33120 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4951896Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:21:02.4952017Z   warnings.warn(
2022-11-23T02:21:02.4952359Z STAGE:2022-11-23 01:58:45 33120:33120 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4952679Z STAGE:2022-11-23 01:58:45 33121:33121 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4953027Z STAGE:2022-11-23 01:58:45 33120:33120 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4953378Z STAGE:2022-11-23 01:58:45 33121:33121 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4953703Z STAGE:2022-11-23 01:58:45 33120:33120 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4954024Z STAGE:2022-11-23 01:58:45 33121:33121 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4954354Z STAGE:2022-11-23 01:58:45 33120:33120 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4954679Z STAGE:2022-11-23 01:58:45 33121:33121 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4955240Z STAGE:2022-11-23 01:58:45 33121:33121 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:58:45 33120:33120 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4955265Z 
2022-11-23T02:21:02.4955432Z ok (6.121s)
2022-11-23T02:21:02.4955452Z 
2022-11-23T02:21:02.4955720Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4955814Z Ran 1 test in 6.121s
2022-11-23T02:21:02.4955834Z 
2022-11-23T02:21:02.4955925Z OK
2022-11-23T02:21:02.4955944Z 
2022-11-23T02:21:02.4956067Z Generating XML reports...
2022-11-23T02:21:02.4956523Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015839.xml
2022-11-23T02:21:02.4956890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4957063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4957480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4957675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4957694Z 
2022-11-23T02:21:02.4957788Z Running tests...
2022-11-23T02:21:02.4958053Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4958364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4958639Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4958859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33235
2022-11-23T02:21:02.4959075Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33236
2022-11-23T02:21:02.4959443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4959615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4959992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4960171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4960534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4960702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4961123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4961317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4961560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4961804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4962206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4962583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4962818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4963045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4963380Z STAGE:2022-11-23 01:58:53 33236:33236 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4964138Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:21:02.4964247Z   warnings.warn(
2022-11-23T02:21:02.4964572Z STAGE:2022-11-23 01:58:53 33235:33235 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4965327Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1506: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:21:02.4965492Z   warnings.warn(
2022-11-23T02:21:02.4965838Z STAGE:2022-11-23 01:58:53 33235:33235 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4966147Z STAGE:2022-11-23 01:58:53 33236:33236 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4966495Z STAGE:2022-11-23 01:58:53 33235:33235 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4966836Z STAGE:2022-11-23 01:58:53 33236:33236 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4967159Z STAGE:2022-11-23 01:58:53 33235:33235 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4967476Z STAGE:2022-11-23 01:58:53 33236:33236 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4967810Z STAGE:2022-11-23 01:58:53 33235:33235 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4968141Z STAGE:2022-11-23 01:58:53 33236:33236 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4968489Z STAGE:2022-11-23 01:58:53 33235:33235 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4968832Z STAGE:2022-11-23 01:58:53 33236:33236 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4968915Z ok (6.038s)
2022-11-23T02:21:02.4968935Z 
2022-11-23T02:21:02.4969196Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4969304Z Ran 1 test in 6.038s
2022-11-23T02:21:02.4969324Z 
2022-11-23T02:21:02.4969413Z OK
2022-11-23T02:21:02.4969432Z 
2022-11-23T02:21:02.4969554Z Generating XML reports...
2022-11-23T02:21:02.4970002Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015848.xml
2022-11-23T02:21:02.4970374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4970548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4970989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4971171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4971191Z 
2022-11-23T02:21:02.4971297Z Running tests...
2022-11-23T02:21:02.4971562Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4971870Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4972131Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4972352Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33350
2022-11-23T02:21:02.4972568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33351
2022-11-23T02:21:02.4972938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4973098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4973479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4973665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4974025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4974196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4974564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4974809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4975057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4975287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4975694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4976087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4976316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4976544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4976875Z STAGE:2022-11-23 01:59:00 33350:33350 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4977198Z STAGE:2022-11-23 01:59:00 33351:33351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4977527Z STAGE:2022-11-23 01:59:00 33350:33350 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4977848Z STAGE:2022-11-23 01:59:00 33351:33351 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4978191Z STAGE:2022-11-23 01:59:00 33350:33350 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4978521Z STAGE:2022-11-23 01:59:00 33351:33351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4978845Z STAGE:2022-11-23 01:59:00 33350:33350 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4979161Z STAGE:2022-11-23 01:59:00 33351:33351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4979708Z STAGE:2022-11-23 01:59:00 33350:33350 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:59:00 33351:33351 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4979731Z 
2022-11-23T02:21:02.4980070Z STAGE:2022-11-23 01:59:00 33350:33350 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4980457Z STAGE:2022-11-23 01:59:00 33351:33351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4980565Z ok (4.260s)
2022-11-23T02:21:02.4980585Z 
2022-11-23T02:21:02.4980850Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4980964Z Ran 1 test in 4.261s
2022-11-23T02:21:02.4980983Z 
2022-11-23T02:21:02.4981055Z OK
2022-11-23T02:21:02.4981074Z 
2022-11-23T02:21:02.4981196Z Generating XML reports...
2022-11-23T02:21:02.4981643Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015856.xml
2022-11-23T02:21:02.4982015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4982189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4982571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4982766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4982786Z 
2022-11-23T02:21:02.4982891Z Running tests...
2022-11-23T02:21:02.4983149Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4983446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4983712Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4983927Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33463
2022-11-23T02:21:02.4984203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33464
2022-11-23T02:21:02.4984574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4984751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4985128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4985317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4985661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4985832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4986197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4986385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4986628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4987172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4987589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4987984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4988211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4988421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4988521Z ok (5.752s)
2022-11-23T02:21:02.4988540Z 
2022-11-23T02:21:02.4988804Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4988918Z Ran 1 test in 5.753s
2022-11-23T02:21:02.4988938Z 
2022-11-23T02:21:02.4989026Z OK
2022-11-23T02:21:02.4989045Z 
2022-11-23T02:21:02.4989166Z Generating XML reports...
2022-11-23T02:21:02.4989615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015903.xml
2022-11-23T02:21:02.4990059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4990227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4990611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4990803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4990823Z 
2022-11-23T02:21:02.4990930Z Running tests...
2022-11-23T02:21:02.4991195Z ----------------------------------------------------------------------
2022-11-23T02:21:02.4991511Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.4991761Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.4991979Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33574
2022-11-23T02:21:02.4992199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33575
2022-11-23T02:21:02.4992553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4992727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4993105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4993293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4993650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.4993892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.4994268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.4994460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.4994688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.4994933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.4995332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4995723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.4995954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.4996286Z STAGE:2022-11-23 01:59:15 33575:33575 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4996514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.4996841Z STAGE:2022-11-23 01:59:15 33574:33574 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4997174Z STAGE:2022-11-23 01:59:15 33574:33574 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4997482Z STAGE:2022-11-23 01:59:15 33575:33575 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4997826Z STAGE:2022-11-23 01:59:15 33574:33574 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4998168Z STAGE:2022-11-23 01:59:15 33575:33575 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.4998704Z STAGE:2022-11-23 01:59:15 33575:33575 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:59:15 33574:33574 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.4998725Z 
2022-11-23T02:21:02.4999054Z STAGE:2022-11-23 01:59:15 33575:33575 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4999427Z STAGE:2022-11-23 01:59:15 33574:33574 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.4999782Z STAGE:2022-11-23 01:59:15 33575:33575 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5000120Z STAGE:2022-11-23 01:59:15 33574:33574 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5000219Z ok (4.219s)
2022-11-23T02:21:02.5000238Z 
2022-11-23T02:21:02.5000497Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5000589Z Ran 1 test in 4.220s
2022-11-23T02:21:02.5000612Z 
2022-11-23T02:21:02.5000702Z OK
2022-11-23T02:21:02.5000721Z 
2022-11-23T02:21:02.5000841Z Generating XML reports...
2022-11-23T02:21:02.5001294Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015912.xml
2022-11-23T02:21:02.5001667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5001842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5002221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5002410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5002430Z 
2022-11-23T02:21:02.5002520Z Running tests...
2022-11-23T02:21:02.5002780Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5003090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5003424Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5003642Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33687
2022-11-23T02:21:02.5003858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33688
2022-11-23T02:21:02.5004233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5004408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5004786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5004959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5005322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5005495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5005865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5006050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5006299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5006540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5006939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5007332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5007545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5007775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5008110Z STAGE:2022-11-23 01:59:22 33688:33688 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5008429Z STAGE:2022-11-23 01:59:22 33687:33687 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5009022Z STAGE:2022-11-23 01:59:22 33688:33688 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:59:22 33687:33687 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5009045Z 
2022-11-23T02:21:02.5009621Z STAGE:2022-11-23 01:59:22 33688:33688 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:59:22 33687:33687 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5009641Z 
2022-11-23T02:21:02.5009966Z STAGE:2022-11-23 01:59:22 33688:33688 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5010291Z STAGE:2022-11-23 01:59:22 33687:33687 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5010620Z STAGE:2022-11-23 01:59:22 33688:33688 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5010945Z STAGE:2022-11-23 01:59:22 33687:33687 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5011272Z STAGE:2022-11-23 01:59:22 33688:33688 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5011614Z STAGE:2022-11-23 01:59:22 33687:33687 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5011712Z ok (4.258s)
2022-11-23T02:21:02.5011731Z 
2022-11-23T02:21:02.5011991Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5012099Z Ran 1 test in 4.259s
2022-11-23T02:21:02.5012118Z 
2022-11-23T02:21:02.5012206Z OK
2022-11-23T02:21:02.5012225Z 
2022-11-23T02:21:02.5012399Z Generating XML reports...
2022-11-23T02:21:02.5012857Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015918.xml
2022-11-23T02:21:02.5013229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5013392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5013775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5013962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5013982Z 
2022-11-23T02:21:02.5014087Z Running tests...
2022-11-23T02:21:02.5014343Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5014656Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5014926Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5015147Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33800
2022-11-23T02:21:02.5015346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33801
2022-11-23T02:21:02.5015719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5015890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5016263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5016452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5016813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5016984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5017356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5017542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5017771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5018112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5018524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5018920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5019147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5019372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5019711Z STAGE:2022-11-23 01:59:29 33801:33801 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5020030Z STAGE:2022-11-23 01:59:29 33800:33800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5020348Z STAGE:2022-11-23 01:59:29 33801:33801 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5020684Z STAGE:2022-11-23 01:59:29 33800:33800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5021253Z STAGE:2022-11-23 01:59:29 33801:33801 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:59:29 33800:33800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5021274Z 
2022-11-23T02:21:02.5021600Z STAGE:2022-11-23 01:59:29 33800:33800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5021914Z STAGE:2022-11-23 01:59:29 33801:33801 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5022306Z STAGE:2022-11-23 01:59:29 33800:33800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5022626Z STAGE:2022-11-23 01:59:29 33801:33801 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5022970Z STAGE:2022-11-23 01:59:29 33800:33800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5023312Z STAGE:2022-11-23 01:59:29 33801:33801 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5023410Z ok (4.249s)
2022-11-23T02:21:02.5023430Z 
2022-11-23T02:21:02.5023673Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5023783Z Ran 1 test in 4.250s
2022-11-23T02:21:02.5023802Z 
2022-11-23T02:21:02.5023890Z OK
2022-11-23T02:21:02.5023909Z 
2022-11-23T02:21:02.5024033Z Generating XML reports...
2022-11-23T02:21:02.5024478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015925.xml
2022-11-23T02:21:02.5024849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5025022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5025401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5025591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5025611Z 
2022-11-23T02:21:02.5025700Z Running tests...
2022-11-23T02:21:02.5025963Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5026270Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5026528Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5026749Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33913
2022-11-23T02:21:02.5027190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33914
2022-11-23T02:21:02.5027573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5027826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5028201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5028395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5028758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5028931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5029298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5029492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5029736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5029981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5030380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5030760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5030990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5031217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5031547Z STAGE:2022-11-23 01:59:37 33914:33914 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5031947Z STAGE:2022-11-23 01:59:37 33913:33913 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5032278Z STAGE:2022-11-23 01:59:37 33913:33913 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5032614Z STAGE:2022-11-23 01:59:37 33914:33914 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5032957Z STAGE:2022-11-23 01:59:37 33913:33913 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5033301Z STAGE:2022-11-23 01:59:37 33914:33914 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5033608Z STAGE:2022-11-23 01:59:37 33913:33913 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5033924Z STAGE:2022-11-23 01:59:37 33914:33914 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5034461Z STAGE:2022-11-23 01:59:39 33913:33913 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:59:39 33914:33914 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5034487Z 
2022-11-23T02:21:02.5035052Z STAGE:2022-11-23 01:59:39 33913:33913 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:59:39 33914:33914 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5035072Z 
2022-11-23T02:21:02.5035395Z STAGE:2022-11-23 01:59:39 33913:33913 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5035711Z STAGE:2022-11-23 01:59:39 33914:33914 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5036039Z STAGE:2022-11-23 01:59:39 33913:33913 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5036364Z STAGE:2022-11-23 01:59:39 33914:33914 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5036710Z STAGE:2022-11-23 01:59:39 33913:33913 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5037048Z STAGE:2022-11-23 01:59:39 33914:33914 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5037131Z ok (7.452s)
2022-11-23T02:21:02.5037167Z 
2022-11-23T02:21:02.5037464Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5037583Z Ran 1 test in 7.452s
2022-11-23T02:21:02.5037602Z 
2022-11-23T02:21:02.5037693Z OK
2022-11-23T02:21:02.5037717Z 
2022-11-23T02:21:02.5037837Z Generating XML reports...
2022-11-23T02:21:02.5038293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015932.xml
2022-11-23T02:21:02.5038662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5038838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5039219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5039393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5039413Z 
2022-11-23T02:21:02.5039519Z Running tests...
2022-11-23T02:21:02.5039781Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5040091Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5040357Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5040574Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34029
2022-11-23T02:21:02.5040790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34030
2022-11-23T02:21:02.5041157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5041371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5041751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5041938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5042300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5042473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5042847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5043031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5043276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5043570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5043963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5044356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5044590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5044818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5045153Z STAGE:2022-11-23 01:59:47 34030:34030 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5045472Z STAGE:2022-11-23 01:59:47 34029:34029 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5045802Z STAGE:2022-11-23 01:59:47 34029:34029 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5046137Z STAGE:2022-11-23 01:59:47 34030:34030 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5046703Z STAGE:2022-11-23 01:59:47 34029:34029 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:59:47 34030:34030 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5046724Z 
2022-11-23T02:21:02.5047100Z STAGE:2022-11-23 01:59:47 34029:34029 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5047415Z STAGE:2022-11-23 01:59:47 34030:34030 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5047956Z STAGE:2022-11-23 01:59:49 34030:34030 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:59:49 34029:34029 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5047976Z 
2022-11-23T02:21:02.5048543Z STAGE:2022-11-23 01:59:49 34029:34029 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:59:49 34030:34030 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5048568Z 
2022-11-23T02:21:02.5049090Z STAGE:2022-11-23 01:59:49 34029:34029 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:59:49 34030:34030 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5049113Z 
2022-11-23T02:21:02.5049651Z STAGE:2022-11-23 01:59:49 34029:34029 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:59:49 34030:34030 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5049671Z 
2022-11-23T02:21:02.5050014Z STAGE:2022-11-23 01:59:49 34029:34029 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5050355Z STAGE:2022-11-23 01:59:49 34030:34030 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5050454Z ok (7.463s)
2022-11-23T02:21:02.5050519Z 
2022-11-23T02:21:02.5050789Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5050901Z Ran 1 test in 7.463s
2022-11-23T02:21:02.5050921Z 
2022-11-23T02:21:02.5051010Z OK
2022-11-23T02:21:02.5051029Z 
2022-11-23T02:21:02.5051135Z Generating XML reports...
2022-11-23T02:21:02.5051588Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015942.xml
2022-11-23T02:21:02.5051956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5052130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5052507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5052696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5052715Z 
2022-11-23T02:21:02.5052821Z Running tests...
2022-11-23T02:21:02.5053085Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5053396Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5053654Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5053876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34145
2022-11-23T02:21:02.5054089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34146
2022-11-23T02:21:02.5054456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5054717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5055153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5055339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5055706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5055859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5056279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5056474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5056718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5056959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5057361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5057799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5058035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5058262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5058581Z STAGE:2022-11-23 01:59:57 34146:34146 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5058901Z STAGE:2022-11-23 01:59:57 34145:34145 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5059228Z STAGE:2022-11-23 01:59:58 34145:34145 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5059549Z STAGE:2022-11-23 01:59:58 34146:34146 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5059887Z STAGE:2022-11-23 01:59:58 34145:34145 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5060229Z STAGE:2022-11-23 01:59:58 34146:34146 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5060820Z STAGE:2022-11-23 01:59:58 34146:34146 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:59:58 34145:34145 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5060841Z 
2022-11-23T02:21:02.5061382Z STAGE:2022-11-23 01:59:59 34146:34146 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 01:59:59 34145:34145 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5061402Z 
2022-11-23T02:21:02.5061965Z STAGE:2022-11-23 01:59:59 34146:34146 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 01:59:59 34145:34145 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5061986Z 
2022-11-23T02:21:02.5062506Z STAGE:2022-11-23 01:59:59 34146:34146 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 01:59:59 34145:34145 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5062529Z 
2022-11-23T02:21:02.5062860Z STAGE:2022-11-23 01:59:59 34146:34146 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5063191Z STAGE:2022-11-23 01:59:59 34145:34145 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5063537Z STAGE:2022-11-23 01:59:59 34146:34146 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5063865Z STAGE:2022-11-23 01:59:59 34145:34145 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5063966Z ok (7.459s)
2022-11-23T02:21:02.5063985Z 
2022-11-23T02:21:02.5064245Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5064356Z Ran 1 test in 7.460s
2022-11-23T02:21:02.5064375Z 
2022-11-23T02:21:02.5064466Z OK
2022-11-23T02:21:02.5064485Z 
2022-11-23T02:21:02.5064606Z Generating XML reports...
2022-11-23T02:21:02.5065059Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015952.xml
2022-11-23T02:21:02.5065425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5065583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5066008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5066206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5066225Z 
2022-11-23T02:21:02.5066332Z Running tests...
2022-11-23T02:21:02.5066597Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5067139Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5067393Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:21:02.5067419Z 
2022-11-23T02:21:02.5067689Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5067800Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5067820Z 
2022-11-23T02:21:02.5067909Z OK (skipped=1)
2022-11-23T02:21:02.5067928Z 
2022-11-23T02:21:02.5068049Z Generating XML reports...
2022-11-23T02:21:02.5068498Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020002.xml
2022-11-23T02:21:02.5068865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5069038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5069412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5069600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5069713Z 
2022-11-23T02:21:02.5069827Z Running tests...
2022-11-23T02:21:02.5070075Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5070392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5070651Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:21:02.5070670Z 
2022-11-23T02:21:02.5070923Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5071032Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5071051Z 
2022-11-23T02:21:02.5071159Z OK (skipped=1)
2022-11-23T02:21:02.5071177Z 
2022-11-23T02:21:02.5071297Z Generating XML reports...
2022-11-23T02:21:02.5071745Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020004.xml
2022-11-23T02:21:02.5072114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5072276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5072652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5072839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5072862Z 
2022-11-23T02:21:02.5072967Z Running tests...
2022-11-23T02:21:02.5073227Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5073536Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5073792Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T02:21:02.5073811Z 
2022-11-23T02:21:02.5074065Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5074174Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5074196Z 
2022-11-23T02:21:02.5074284Z OK (skipped=1)
2022-11-23T02:21:02.5074302Z 
2022-11-23T02:21:02.5074423Z Generating XML reports...
2022-11-23T02:21:02.5074868Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020007.xml
2022-11-23T02:21:02.5075298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5075484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5075866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5076057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5076077Z 
2022-11-23T02:21:02.5076183Z Running tests...
2022-11-23T02:21:02.5076426Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5076736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5077008Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T02:21:02.5077027Z 
2022-11-23T02:21:02.5077284Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5077393Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5077416Z 
2022-11-23T02:21:02.5077522Z OK (skipped=1)
2022-11-23T02:21:02.5077541Z 
2022-11-23T02:21:02.5077661Z Generating XML reports...
2022-11-23T02:21:02.5078106Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020009.xml
2022-11-23T02:21:02.5078475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5078632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5079008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5079256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5079275Z 
2022-11-23T02:21:02.5079383Z Running tests...
2022-11-23T02:21:02.5079643Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5079957Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5080217Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:21:02.5080237Z 
2022-11-23T02:21:02.5080494Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5080603Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5080621Z 
2022-11-23T02:21:02.5080709Z OK (skipped=1)
2022-11-23T02:21:02.5080728Z 
2022-11-23T02:21:02.5080850Z Generating XML reports...
2022-11-23T02:21:02.5081296Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020012.xml
2022-11-23T02:21:02.5081666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5081840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5082217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5082407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5082425Z 
2022-11-23T02:21:02.5082530Z Running tests...
2022-11-23T02:21:02.5082791Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5083087Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5083359Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-11-23T02:21:02.5083382Z 
2022-11-23T02:21:02.5083637Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5083746Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5083765Z 
2022-11-23T02:21:02.5083869Z OK (skipped=1)
2022-11-23T02:21:02.5083887Z 
2022-11-23T02:21:02.5084007Z Generating XML reports...
2022-11-23T02:21:02.5084503Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020014.xml
2022-11-23T02:21:02.5084885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5085042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5085418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5085606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5085625Z 
2022-11-23T02:21:02.5085734Z Running tests...
2022-11-23T02:21:02.5085994Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5086301Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5086551Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-11-23T02:21:02.5086575Z 
2022-11-23T02:21:02.5086831Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5086940Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5086959Z 
2022-11-23T02:21:02.5087047Z OK (skipped=1)
2022-11-23T02:21:02.5087083Z 
2022-11-23T02:21:02.5087187Z Generating XML reports...
2022-11-23T02:21:02.5087633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020016.xml
2022-11-23T02:21:02.5087999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5088229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5088608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5088796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5088816Z 
2022-11-23T02:21:02.5088926Z Running tests...
2022-11-23T02:21:02.5089186Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5089479Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5089754Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:21:02.5089773Z 
2022-11-23T02:21:02.5090025Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5090131Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5090150Z 
2022-11-23T02:21:02.5090260Z OK (skipped=1)
2022-11-23T02:21:02.5090279Z 
2022-11-23T02:21:02.5090400Z Generating XML reports...
2022-11-23T02:21:02.5090843Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020019.xml
2022-11-23T02:21:02.5091212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5091385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5091747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5091935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5091955Z 
2022-11-23T02:21:02.5092059Z Running tests...
2022-11-23T02:21:02.5092317Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5092624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5092912Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:21:02.5092931Z 
2022-11-23T02:21:02.5093187Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5093295Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5093314Z 
2022-11-23T02:21:02.5093451Z OK (skipped=1)
2022-11-23T02:21:02.5093489Z 
2022-11-23T02:21:02.5093597Z Generating XML reports...
2022-11-23T02:21:02.5094046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020021.xml
2022-11-23T02:21:02.5094412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5094586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5094962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5095153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5095172Z 
2022-11-23T02:21:02.5095276Z Running tests...
2022-11-23T02:21:02.5095535Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5095830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5096124Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:21:02.5096144Z 
2022-11-23T02:21:02.5096399Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5096506Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5096525Z 
2022-11-23T02:21:02.5096628Z OK (skipped=1)
2022-11-23T02:21:02.5096646Z 
2022-11-23T02:21:02.5096764Z Generating XML reports...
2022-11-23T02:21:02.5097209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020024.xml
2022-11-23T02:21:02.5097635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5097812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5098175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5098366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5098385Z 
2022-11-23T02:21:02.5098489Z Running tests...
2022-11-23T02:21:02.5098746Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5099054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5099343Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:21:02.5099366Z 
2022-11-23T02:21:02.5099623Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5099731Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5099750Z 
2022-11-23T02:21:02.5099855Z OK (skipped=1)
2022-11-23T02:21:02.5099874Z 
2022-11-23T02:21:02.5099977Z Generating XML reports...
2022-11-23T02:21:02.5100425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020026.xml
2022-11-23T02:21:02.5100789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5100960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5101335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5101523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5101546Z 
2022-11-23T02:21:02.5101651Z Running tests...
2022-11-23T02:21:02.5101910Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5102202Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5102556Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:21:02.5102578Z 
2022-11-23T02:21:02.5102843Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5102953Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5102972Z 
2022-11-23T02:21:02.5103076Z OK (skipped=1)
2022-11-23T02:21:02.5103095Z 
2022-11-23T02:21:02.5103215Z Generating XML reports...
2022-11-23T02:21:02.5103654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020028.xml
2022-11-23T02:21:02.5104015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5104190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5104549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5104736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5104759Z 
2022-11-23T02:21:02.5104865Z Running tests...
2022-11-23T02:21:02.5105123Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5105431Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5105726Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:21:02.5105746Z 
2022-11-23T02:21:02.5106000Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5106164Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5106183Z 
2022-11-23T02:21:02.5106289Z OK (skipped=1)
2022-11-23T02:21:02.5106308Z 
2022-11-23T02:21:02.5106412Z Generating XML reports...
2022-11-23T02:21:02.5107170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020031.xml
2022-11-23T02:21:02.5107574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5107749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5108127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5108315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5108334Z 
2022-11-23T02:21:02.5108442Z Running tests...
2022-11-23T02:21:02.5108701Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5109016Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5109302Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:21:02.5109340Z 
2022-11-23T02:21:02.5109584Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5109694Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5109713Z 
2022-11-23T02:21:02.5109816Z OK (skipped=1)
2022-11-23T02:21:02.5109834Z 
2022-11-23T02:21:02.5109955Z Generating XML reports...
2022-11-23T02:21:02.5110399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020033.xml
2022-11-23T02:21:02.5110765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5110936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5111314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5111486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5111506Z 
2022-11-23T02:21:02.5111610Z Running tests...
2022-11-23T02:21:02.5111952Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5112278Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5112571Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:21:02.5112590Z 
2022-11-23T02:21:02.5112847Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5112958Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5112977Z 
2022-11-23T02:21:02.5113083Z OK (skipped=1)
2022-11-23T02:21:02.5113102Z 
2022-11-23T02:21:02.5113224Z Generating XML reports...
2022-11-23T02:21:02.5113657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020036.xml
2022-11-23T02:21:02.5114022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5114198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5114574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5114759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5114779Z 
2022-11-23T02:21:02.5114885Z Running tests...
2022-11-23T02:21:02.5115144Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5115451Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5115731Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:21:02.5115834Z 
2022-11-23T02:21:02.5116083Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5116193Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5116212Z 
2022-11-23T02:21:02.5116319Z OK (skipped=1)
2022-11-23T02:21:02.5116338Z 
2022-11-23T02:21:02.5116463Z Generating XML reports...
2022-11-23T02:21:02.5116907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020038.xml
2022-11-23T02:21:02.5117275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5117448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5117819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5117996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5118031Z 
2022-11-23T02:21:02.5118121Z Running tests...
2022-11-23T02:21:02.5118379Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5118688Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5118978Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:21:02.5118997Z 
2022-11-23T02:21:02.5119252Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5119361Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5119380Z 
2022-11-23T02:21:02.5119484Z OK (skipped=1)
2022-11-23T02:21:02.5119503Z 
2022-11-23T02:21:02.5119622Z Generating XML reports...
2022-11-23T02:21:02.5120044Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020040.xml
2022-11-23T02:21:02.5120416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5120586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5120959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5121202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5121223Z 
2022-11-23T02:21:02.5121336Z Running tests...
2022-11-23T02:21:02.5121596Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5121903Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5122200Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:21:02.5122220Z 
2022-11-23T02:21:02.5122459Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5122573Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5122591Z 
2022-11-23T02:21:02.5122694Z OK (skipped=1)
2022-11-23T02:21:02.5122713Z 
2022-11-23T02:21:02.5122833Z Generating XML reports...
2022-11-23T02:21:02.5123285Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020043.xml
2022-11-23T02:21:02.5123654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5123827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5124203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5124391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5124410Z 
2022-11-23T02:21:02.5124499Z Running tests...
2022-11-23T02:21:02.5124756Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5125126Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5125418Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:21:02.5125437Z 
2022-11-23T02:21:02.5125694Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5125803Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5125822Z 
2022-11-23T02:21:02.5125930Z OK (skipped=1)
2022-11-23T02:21:02.5125949Z 
2022-11-23T02:21:02.5126071Z Generating XML reports...
2022-11-23T02:21:02.5126497Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020045.xml
2022-11-23T02:21:02.5126864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5127039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5127411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5127598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5127617Z 
2022-11-23T02:21:02.5127722Z Running tests...
2022-11-23T02:21:02.5127983Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5128293Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5128597Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:21:02.5128617Z 
2022-11-23T02:21:02.5128855Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5128964Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5128983Z 
2022-11-23T02:21:02.5129090Z OK (skipped=1)
2022-11-23T02:21:02.5129109Z 
2022-11-23T02:21:02.5129227Z Generating XML reports...
2022-11-23T02:21:02.5129670Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020047.xml
2022-11-23T02:21:02.5130083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5130264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5130643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5130829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5130848Z 
2022-11-23T02:21:02.5130937Z Running tests...
2022-11-23T02:21:02.5131192Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5131499Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5131802Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:21:02.5131821Z 
2022-11-23T02:21:02.5132077Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5132185Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5132204Z 
2022-11-23T02:21:02.5132311Z OK (skipped=1)
2022-11-23T02:21:02.5132330Z 
2022-11-23T02:21:02.5132451Z Generating XML reports...
2022-11-23T02:21:02.5132899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020050.xml
2022-11-23T02:21:02.5133250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5133424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5133801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5134058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5134077Z 
2022-11-23T02:21:02.5134183Z Running tests...
2022-11-23T02:21:02.5134446Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5134758Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5135067Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:21:02.5135088Z 
2022-11-23T02:21:02.5135345Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5135437Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5135456Z 
2022-11-23T02:21:02.5135560Z OK (skipped=1)
2022-11-23T02:21:02.5135579Z 
2022-11-23T02:21:02.5135698Z Generating XML reports...
2022-11-23T02:21:02.5136141Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020052.xml
2022-11-23T02:21:02.5136515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5136688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5137065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5137253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5137273Z 
2022-11-23T02:21:02.5137378Z Running tests...
2022-11-23T02:21:02.5137623Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5137929Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5138219Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-11-23T02:21:02.5138242Z 
2022-11-23T02:21:02.5138496Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5138603Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5138622Z 
2022-11-23T02:21:02.5138724Z OK (skipped=1)
2022-11-23T02:21:02.5138743Z 
2022-11-23T02:21:02.5138863Z Generating XML reports...
2022-11-23T02:21:02.5139358Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020055.xml
2022-11-23T02:21:02.5139721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5139896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5140270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5140458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5140481Z 
2022-11-23T02:21:02.5140586Z Running tests...
2022-11-23T02:21:02.5140847Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5141156Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5141459Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-11-23T02:21:02.5141479Z 
2022-11-23T02:21:02.5141734Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5141827Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5141845Z 
2022-11-23T02:21:02.5141952Z OK (skipped=1)
2022-11-23T02:21:02.5141971Z 
2022-11-23T02:21:02.5142090Z Generating XML reports...
2022-11-23T02:21:02.5142532Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020057.xml
2022-11-23T02:21:02.5142896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5143126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5143508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5143701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5143721Z 
2022-11-23T02:21:02.5143828Z Running tests...
2022-11-23T02:21:02.5144067Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5144374Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5144635Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5144853Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35053
2022-11-23T02:21:02.5145067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35054
2022-11-23T02:21:02.5145438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5145612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5145990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5146162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5146525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5146694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5147307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5147496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5147749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5147996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5148482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5148897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5149110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5149334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5149572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.5149811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.5150210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5150601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5150700Z ok (6.977s)
2022-11-23T02:21:02.5150724Z 
2022-11-23T02:21:02.5150987Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5151097Z Ran 1 test in 6.977s
2022-11-23T02:21:02.5151116Z 
2022-11-23T02:21:02.5151188Z OK
2022-11-23T02:21:02.5151207Z 
2022-11-23T02:21:02.5151328Z Generating XML reports...
2022-11-23T02:21:02.5151778Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020059.xml
2022-11-23T02:21:02.5152146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5152395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5152779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5152968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5152987Z 
2022-11-23T02:21:02.5153097Z Running tests...
2022-11-23T02:21:02.5153340Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5153654Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5153916Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5154132Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35174
2022-11-23T02:21:02.5154347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35175
2022-11-23T02:21:02.5154711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5154887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5155262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5155454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5155796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5155967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5156333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5156523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5156765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5157013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5157413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5157898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5158136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5158346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5158500Z skip: Need at least 3 CUDA devices (4.147s)
2022-11-23T02:21:02.5158519Z 
2022-11-23T02:21:02.5158786Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5158894Z Ran 1 test in 4.147s
2022-11-23T02:21:02.5158913Z 
2022-11-23T02:21:02.5159022Z OK (skipped=1)
2022-11-23T02:21:02.5159041Z 
2022-11-23T02:21:02.5159163Z Generating XML reports...
2022-11-23T02:21:02.5159610Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020109.xml
2022-11-23T02:21:02.5159974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5160134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5160514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5160700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5160719Z 
2022-11-23T02:21:02.5160824Z Running tests...
2022-11-23T02:21:02.5161082Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5161390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5161751Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-11-23T02:21:02.5161771Z 
2022-11-23T02:21:02.5162035Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5162144Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5162163Z 
2022-11-23T02:21:02.5162252Z OK (skipped=1)
2022-11-23T02:21:02.5162274Z 
2022-11-23T02:21:02.5162396Z Generating XML reports...
2022-11-23T02:21:02.5162843Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020116.xml
2022-11-23T02:21:02.5163212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5163383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5163758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5163949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5163969Z 
2022-11-23T02:21:02.5164072Z Running tests...
2022-11-23T02:21:02.5164317Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5164632Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5164870Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5165087Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35316
2022-11-23T02:21:02.5165303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35317
2022-11-23T02:21:02.5165671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5165842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5166214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5166404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5166747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5166966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5167346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5167531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5167778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5168022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5168418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5168819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5169051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5169265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5169367Z ok (5.026s)
2022-11-23T02:21:02.5169387Z 
2022-11-23T02:21:02.5169649Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5169757Z Ran 1 test in 5.027s
2022-11-23T02:21:02.5169777Z 
2022-11-23T02:21:02.5169866Z OK
2022-11-23T02:21:02.5169885Z 
2022-11-23T02:21:02.5170005Z Generating XML reports...
2022-11-23T02:21:02.5170456Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020118.xml
2022-11-23T02:21:02.5170888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5171046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5171424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5171616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5171635Z 
2022-11-23T02:21:02.5171742Z Running tests...
2022-11-23T02:21:02.5172004Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5172314Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5172563Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5172782Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35425
2022-11-23T02:21:02.5173003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35426
2022-11-23T02:21:02.5173356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5173529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5173907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5174096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5174454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5174623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5174993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5175178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5175410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5175652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5176093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5176496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5176726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5176954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5177054Z ok (6.622s)
2022-11-23T02:21:02.5177074Z 
2022-11-23T02:21:02.5177336Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5177449Z Ran 1 test in 6.623s
2022-11-23T02:21:02.5177469Z 
2022-11-23T02:21:02.5177541Z OK
2022-11-23T02:21:02.5177560Z 
2022-11-23T02:21:02.5177680Z Generating XML reports...
2022-11-23T02:21:02.5178130Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020125.xml
2022-11-23T02:21:02.5178498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5178672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5179047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5179235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5179255Z 
2022-11-23T02:21:02.5179360Z Running tests...
2022-11-23T02:21:02.5179602Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5179972Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5180233Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5180451Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35536
2022-11-23T02:21:02.5180673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35537
2022-11-23T02:21:02.5181038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5181212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5181586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5181773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5182117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5182291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5182660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5182850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5183094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5183337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5183735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5184126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5184354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5184569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5184811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.5185095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.5185503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5185891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5185992Z ok (5.049s)
2022-11-23T02:21:02.5186012Z 
2022-11-23T02:21:02.5186274Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5186384Z Ran 1 test in 5.049s
2022-11-23T02:21:02.5186403Z 
2022-11-23T02:21:02.5186480Z OK
2022-11-23T02:21:02.5186515Z 
2022-11-23T02:21:02.5186619Z Generating XML reports...
2022-11-23T02:21:02.5187299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020135.xml
2022-11-23T02:21:02.5187682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5187861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5188241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5188430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5188449Z 
2022-11-23T02:21:02.5188556Z Running tests...
2022-11-23T02:21:02.5188812Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5189106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5189466Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5189685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35651
2022-11-23T02:21:02.5189906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35652
2022-11-23T02:21:02.5190282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5190457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5190831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5191020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5191363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5191539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5191912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5192097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5192344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5192585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5192983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5193375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5193602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5193817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5193976Z skip: Skipped due to small world size. (4.255s)
2022-11-23T02:21:02.5193996Z 
2022-11-23T02:21:02.5194259Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5194369Z Ran 1 test in 4.255s
2022-11-23T02:21:02.5194388Z 
2022-11-23T02:21:02.5194553Z OK (skipped=1)
2022-11-23T02:21:02.5194575Z 
2022-11-23T02:21:02.5194702Z Generating XML reports...
2022-11-23T02:21:02.5195159Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020142.xml
2022-11-23T02:21:02.5195528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5195700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5196060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5196255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5196274Z 
2022-11-23T02:21:02.5196378Z Running tests...
2022-11-23T02:21:02.5196637Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5196951Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5197201Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5197418Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35760
2022-11-23T02:21:02.5197632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35761
2022-11-23T02:21:02.5197984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5198159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5198613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5198802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5199160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5199337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5199707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5199893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5200135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5200361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5200760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5201160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5201391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5201623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5201781Z skip: Skipped due to small world size. (4.176s)
2022-11-23T02:21:02.5201800Z 
2022-11-23T02:21:02.5202063Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5202172Z Ran 1 test in 4.177s
2022-11-23T02:21:02.5202191Z 
2022-11-23T02:21:02.5202295Z OK (skipped=1)
2022-11-23T02:21:02.5202314Z 
2022-11-23T02:21:02.5202418Z Generating XML reports...
2022-11-23T02:21:02.5202866Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020149.xml
2022-11-23T02:21:02.5203238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5203410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5203834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5204032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5204052Z 
2022-11-23T02:21:02.5204160Z Running tests...
2022-11-23T02:21:02.5204422Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5204716Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5204975Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5205192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35869
2022-11-23T02:21:02.5205415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35870
2022-11-23T02:21:02.5205783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5205959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5206338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5206524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5206889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5207042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5207411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5207651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5207898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5208142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5208547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5208945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5209173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5209398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5209538Z skip: Skipped due to small world size. (4.259s)
2022-11-23T02:21:02.5209561Z 
2022-11-23T02:21:02.5209826Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5209935Z Ran 1 test in 4.260s
2022-11-23T02:21:02.5209955Z 
2022-11-23T02:21:02.5210059Z OK (skipped=1)
2022-11-23T02:21:02.5210078Z 
2022-11-23T02:21:02.5210198Z Generating XML reports...
2022-11-23T02:21:02.5210650Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020156.xml
2022-11-23T02:21:02.5211020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5211193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5211553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5211743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5211762Z 
2022-11-23T02:21:02.5211872Z Running tests...
2022-11-23T02:21:02.5212134Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5212446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5212714Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5212976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35978
2022-11-23T02:21:02.5213199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35979
2022-11-23T02:21:02.5213573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5213730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5214105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5214293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5214657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5214828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5215202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5215388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5215632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5215856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5216259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5216652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5216931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5217158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5217400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.5217640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.5218035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5218422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5218505Z ok (5.280s)
2022-11-23T02:21:02.5218540Z 
2022-11-23T02:21:02.5218788Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5218901Z Ran 1 test in 5.280s
2022-11-23T02:21:02.5218920Z 
2022-11-23T02:21:02.5219010Z OK
2022-11-23T02:21:02.5219030Z 
2022-11-23T02:21:02.5219151Z Generating XML reports...
2022-11-23T02:21:02.5219600Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020203.xml
2022-11-23T02:21:02.5219970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5220144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5220523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5220695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5220714Z 
2022-11-23T02:21:02.5220819Z Running tests...
2022-11-23T02:21:02.5221075Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5221389Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5221651Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5221868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36093
2022-11-23T02:21:02.5222129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36094
2022-11-23T02:21:02.5222513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5222670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5223047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5223233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5223602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5223771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5224141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5224331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5224576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5224817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5225202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5225596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5225876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5226104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5226343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5226591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5227203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5227609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5227711Z ok (5.320s)
2022-11-23T02:21:02.5227731Z 
2022-11-23T02:21:02.5227979Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5228087Z Ran 1 test in 5.320s
2022-11-23T02:21:02.5228112Z 
2022-11-23T02:21:02.5228200Z OK
2022-11-23T02:21:02.5228219Z 
2022-11-23T02:21:02.5228339Z Generating XML reports...
2022-11-23T02:21:02.5228790Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020210.xml
2022-11-23T02:21:02.5229162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5229337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5229713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5229884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5229921Z 
2022-11-23T02:21:02.5230010Z Running tests...
2022-11-23T02:21:02.5230271Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5230582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5230847Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5231063Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36208
2022-11-23T02:21:02.5231355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36209
2022-11-23T02:21:02.5231746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5231917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5232281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5232470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5232832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5233007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5233379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5233566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5233814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5234055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5234437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5234870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5235105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5235402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5235560Z skip: Skipped due to small world size. (4.224s)
2022-11-23T02:21:02.5235580Z 
2022-11-23T02:21:02.5235848Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5235958Z Ran 1 test in 4.225s
2022-11-23T02:21:02.5235981Z 
2022-11-23T02:21:02.5236088Z OK (skipped=1)
2022-11-23T02:21:02.5236108Z 
2022-11-23T02:21:02.5236230Z Generating XML reports...
2022-11-23T02:21:02.5236663Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020218.xml
2022-11-23T02:21:02.5237034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5237207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5237581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5237773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5237793Z 
2022-11-23T02:21:02.5237898Z Running tests...
2022-11-23T02:21:02.5238160Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5238474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5238741Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5238941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36317
2022-11-23T02:21:02.5239156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36318
2022-11-23T02:21:02.5239523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5239696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5240075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5240264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5240678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5240858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5241218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5241405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5241649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5241892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5242294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5242688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5242919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5243144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5243244Z ok (4.253s)
2022-11-23T02:21:02.5243264Z 
2022-11-23T02:21:02.5243511Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5243619Z Ran 1 test in 4.253s
2022-11-23T02:21:02.5243641Z 
2022-11-23T02:21:02.5243729Z OK
2022-11-23T02:21:02.5243748Z 
2022-11-23T02:21:02.5243867Z Generating XML reports...
2022-11-23T02:21:02.5244312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020225.xml
2022-11-23T02:21:02.5244753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5244927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5245303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5245496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5245515Z 
2022-11-23T02:21:02.5245604Z Running tests...
2022-11-23T02:21:02.5245866Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5246174Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5246443Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5246664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36426
2022-11-23T02:21:02.5246879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36427
2022-11-23T02:21:02.5247246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5247423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5247786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5247978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5248335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5248505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5248875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5249064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5249311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5249553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5250013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5250402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5250631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5250859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5250958Z ok (4.203s)
2022-11-23T02:21:02.5250977Z 
2022-11-23T02:21:02.5251244Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5251353Z Ran 1 test in 4.203s
2022-11-23T02:21:02.5251373Z 
2022-11-23T02:21:02.5251462Z OK
2022-11-23T02:21:02.5251481Z 
2022-11-23T02:21:02.5251602Z Generating XML reports...
2022-11-23T02:21:02.5252039Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020232.xml
2022-11-23T02:21:02.5252410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5252585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5252962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5253151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5253170Z 
2022-11-23T02:21:02.5253276Z Running tests...
2022-11-23T02:21:02.5253595Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5253907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5254180Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:21:02.5254199Z 
2022-11-23T02:21:02.5254445Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5254555Z Ran 1 test in 0.003s
2022-11-23T02:21:02.5254574Z 
2022-11-23T02:21:02.5254682Z OK (skipped=1)
2022-11-23T02:21:02.5254701Z 
2022-11-23T02:21:02.5254820Z Generating XML reports...
2022-11-23T02:21:02.5255267Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020238.xml
2022-11-23T02:21:02.5255635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5255814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5256189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5256377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5256396Z 
2022-11-23T02:21:02.5256484Z Running tests...
2022-11-23T02:21:02.5256747Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5257062Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5257316Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-11-23T02:21:02.5257336Z 
2022-11-23T02:21:02.5257632Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5257745Z Ran 1 test in 0.003s
2022-11-23T02:21:02.5257764Z 
2022-11-23T02:21:02.5257872Z OK (skipped=1)
2022-11-23T02:21:02.5257894Z 
2022-11-23T02:21:02.5258021Z Generating XML reports...
2022-11-23T02:21:02.5258457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020241.xml
2022-11-23T02:21:02.5258825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5259049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5259433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5259623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5259642Z 
2022-11-23T02:21:02.5259750Z Running tests...
2022-11-23T02:21:02.5260006Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5260316Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5260589Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:21:02.5260608Z 
2022-11-23T02:21:02.5260848Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5260957Z Ran 1 test in 0.003s
2022-11-23T02:21:02.5260976Z 
2022-11-23T02:21:02.5261080Z OK (skipped=1)
2022-11-23T02:21:02.5261102Z 
2022-11-23T02:21:02.5261223Z Generating XML reports...
2022-11-23T02:21:02.5261671Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020243.xml
2022-11-23T02:21:02.5262036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5262210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5262582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5262843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5262862Z 
2022-11-23T02:21:02.5262951Z Running tests...
2022-11-23T02:21:02.5263217Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5263525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5263786Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:21:02.5263805Z 
2022-11-23T02:21:02.5264063Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5264172Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5264191Z 
2022-11-23T02:21:02.5264296Z OK (skipped=1)
2022-11-23T02:21:02.5264316Z 
2022-11-23T02:21:02.5264434Z Generating XML reports...
2022-11-23T02:21:02.5264862Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020246.xml
2022-11-23T02:21:02.5265234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5265409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5265782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5265976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5265996Z 
2022-11-23T02:21:02.5266103Z Running tests...
2022-11-23T02:21:02.5266361Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5266669Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5267161Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:21:02.5267183Z 
2022-11-23T02:21:02.5267440Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5267557Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5267577Z 
2022-11-23T02:21:02.5267681Z OK (skipped=1)
2022-11-23T02:21:02.5267700Z 
2022-11-23T02:21:02.5267819Z Generating XML reports...
2022-11-23T02:21:02.5268344Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020248.xml
2022-11-23T02:21:02.5268730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5268905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5269281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5269470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5269489Z 
2022-11-23T02:21:02.5269578Z Running tests...
2022-11-23T02:21:02.5269840Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5270153Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5270425Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:21:02.5270444Z 
2022-11-23T02:21:02.5270702Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5270810Z Ran 1 test in 0.003s
2022-11-23T02:21:02.5270829Z 
2022-11-23T02:21:02.5270933Z OK (skipped=1)
2022-11-23T02:21:02.5270952Z 
2022-11-23T02:21:02.5271070Z Generating XML reports...
2022-11-23T02:21:02.5271514Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020250.xml
2022-11-23T02:21:02.5271866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5272039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5272495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5272683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5272702Z 
2022-11-23T02:21:02.5272810Z Running tests...
2022-11-23T02:21:02.5273073Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5273383Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5273643Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:21:02.5273662Z 
2022-11-23T02:21:02.5273916Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5274008Z Ran 1 test in 0.003s
2022-11-23T02:21:02.5274027Z 
2022-11-23T02:21:02.5274131Z OK (skipped=1)
2022-11-23T02:21:02.5274150Z 
2022-11-23T02:21:02.5274272Z Generating XML reports...
2022-11-23T02:21:02.5274715Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020253.xml
2022-11-23T02:21:02.5275079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5275256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5275626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5275814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5275833Z 
2022-11-23T02:21:02.5275923Z Running tests...
2022-11-23T02:21:02.5276184Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5276490Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5276750Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-11-23T02:21:02.5276773Z 
2022-11-23T02:21:02.5277032Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5277141Z Ran 1 test in 0.002s
2022-11-23T02:21:02.5277160Z 
2022-11-23T02:21:02.5277264Z OK (skipped=1)
2022-11-23T02:21:02.5277283Z 
2022-11-23T02:21:02.5277453Z Generating XML reports...
2022-11-23T02:21:02.5277903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020255.xml
2022-11-23T02:21:02.5278253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5278427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5278793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5278986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5279004Z 
2022-11-23T02:21:02.5279111Z Running tests...
2022-11-23T02:21:02.5279368Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5279675Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5279921Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5280122Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36799
2022-11-23T02:21:02.5280337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36800
2022-11-23T02:21:02.5280706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5280877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5281247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5281488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5281856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5282033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5282404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5282571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5282816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5283059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5283456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5283855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5284082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5284311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5284649Z STAGE:2022-11-23 02:03:01 36799:36799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5284970Z STAGE:2022-11-23 02:03:01 36800:36800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5285289Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5285610Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5285951Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5286299Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5286621Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5286993Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5287337Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5287660Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5287999Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5288325Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5288655Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5288971Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5289300Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5289644Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5289972Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5290309Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5290631Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5290949Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5291327Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5291653Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5291993Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5292334Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5292657Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5292976Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5293305Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5293642Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5293975Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5294299Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5294624Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5294945Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5295272Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5295592Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5295929Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5296269Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5296593Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5296915Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5297276Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5297629Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5297959Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5298297Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5298619Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5298936Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5299267Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5299588Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5299931Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5300258Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5300581Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5300964Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5301523Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5301980Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5302295Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5302689Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5303055Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5303422Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5303789Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5304158Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5304536Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5304973Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5305357Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5305668Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5306037Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5306402Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5307013Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5307424Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5307789Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5308209Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5308639Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5309098Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5309441Z STAGE:2022-11-23 02:03:02 36800:36800 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5309824Z STAGE:2022-11-23 02:03:02 36799:36799 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5309962Z ok (4.276s)
2022-11-23T02:21:02.5309983Z 
2022-11-23T02:21:02.5310282Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5310429Z Ran 1 test in 4.276s
2022-11-23T02:21:02.5310449Z 
2022-11-23T02:21:02.5310587Z OK
2022-11-23T02:21:02.5310606Z 
2022-11-23T02:21:02.5310769Z Generating XML reports...
2022-11-23T02:21:02.5311310Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020258.xml
2022-11-23T02:21:02.5311675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5311896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5312318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5312549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5312569Z 
2022-11-23T02:21:02.5312715Z Running tests...
2022-11-23T02:21:02.5313023Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5313430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5313727Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5314661Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2022-11-23T02:21:02.5314683Z 
2022-11-23T02:21:02.5314990Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5315087Z Ran 1 test in 1.641s
2022-11-23T02:21:02.5315106Z 
2022-11-23T02:21:02.5315249Z OK (skipped=1)
2022-11-23T02:21:02.5315269Z 
2022-11-23T02:21:02.5315428Z Generating XML reports...
2022-11-23T02:21:02.5315952Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020304.xml
2022-11-23T02:21:02.5316366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5316583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5316999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5317263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5317287Z 
2022-11-23T02:21:02.5317435Z Running tests...
2022-11-23T02:21:02.5317685Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5318036Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5318347Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5318607Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36946
2022-11-23T02:21:02.5318907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36947
2022-11-23T02:21:02.5319325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5319537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5319997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5320233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5320667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5320876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5321290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5321516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5321802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5322091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5322532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5323018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5323243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5323508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5323784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.5324062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.5324611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5325050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5325429Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5325811Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5326227Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5326547Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5326932Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5327317Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5327695Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5328058Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5328435Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5328811Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5329194Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5329609Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5329927Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5330285Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5330658Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5331085Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5331514Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5331907Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5332270Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5332633Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5333043Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5333370Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5333749Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5334124Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5334508Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5334867Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5335237Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5335603Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5335984Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5336462Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5336781Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5337196Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5337579Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5337945Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5338324Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5338702Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5339065Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5339429Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5339832Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5340171Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5340550Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5340926Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5341289Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5341646Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5342013Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5342385Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5342768Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5343292Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5343683Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5344046Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5344415Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5344794Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5345165Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5345546Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5345920Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5346283Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5346696Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5347271Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5347658Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5348041Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5348510Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5348873Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5349255Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5349640Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5350063Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5350494Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5350813Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5351170Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5351545Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5351910Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5352307Z STAGE:2022-11-23 02:03:13 36947:36947 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5352693Z STAGE:2022-11-23 02:03:13 36946:36946 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5352829Z ok (4.336s)
2022-11-23T02:21:02.5352851Z 
2022-11-23T02:21:02.5353150Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5353247Z Ran 1 test in 4.336s
2022-11-23T02:21:02.5353352Z 
2022-11-23T02:21:02.5353431Z OK
2022-11-23T02:21:02.5353450Z 
2022-11-23T02:21:02.5353608Z Generating XML reports...
2022-11-23T02:21:02.5354106Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020309.xml
2022-11-23T02:21:02.5354535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5354749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5355232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5355515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5355535Z 
2022-11-23T02:21:02.5355679Z Running tests...
2022-11-23T02:21:02.5355929Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5356318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5356624Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5356879Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37065
2022-11-23T02:21:02.5357139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37066
2022-11-23T02:21:02.5357595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5357814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5358238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5358413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5358818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5359075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5359495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5359782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5360063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5360348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5360838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5361279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5361496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5361771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5362006Z skip: Skipped due to small world size. (4.207s)
2022-11-23T02:21:02.5362031Z 
2022-11-23T02:21:02.5362338Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5362486Z Ran 1 test in 4.208s
2022-11-23T02:21:02.5362506Z 
2022-11-23T02:21:02.5362648Z OK (skipped=1)
2022-11-23T02:21:02.5362667Z 
2022-11-23T02:21:02.5362824Z Generating XML reports...
2022-11-23T02:21:02.5363315Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020315.xml
2022-11-23T02:21:02.5363740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5363903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5364319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5364583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5364603Z 
2022-11-23T02:21:02.5364753Z Running tests...
2022-11-23T02:21:02.5365056Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5365406Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5365706Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5366073Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37174
2022-11-23T02:21:02.5366281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37175
2022-11-23T02:21:02.5366696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5366905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5367362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5367592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5368002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5368223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5368638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5368865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5369096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5369374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5369812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5370288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5370618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5370893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5371769Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:21:02.5371925Z   warnings.warn(
2022-11-23T02:21:02.5372730Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1402: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-11-23T02:21:02.5372879Z   warnings.warn(
2022-11-23T02:21:02.5372963Z ok (5.644s)
2022-11-23T02:21:02.5372983Z 
2022-11-23T02:21:02.5373284Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5373468Z Ran 1 test in 5.645s
2022-11-23T02:21:02.5373487Z 
2022-11-23T02:21:02.5373623Z OK
2022-11-23T02:21:02.5373642Z 
2022-11-23T02:21:02.5373808Z Generating XML reports...
2022-11-23T02:21:02.5374308Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020322.xml
2022-11-23T02:21:02.5374714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5374927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5375289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5375516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5375540Z 
2022-11-23T02:21:02.5375682Z Running tests...
2022-11-23T02:21:02.5376029Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5376385Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5376784Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5377577Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.641s)
2022-11-23T02:21:02.5377598Z 
2022-11-23T02:21:02.5377906Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5378056Z Ran 1 test in 1.641s
2022-11-23T02:21:02.5378075Z 
2022-11-23T02:21:02.5378233Z OK (skipped=1)
2022-11-23T02:21:02.5378254Z 
2022-11-23T02:21:02.5378360Z Generating XML reports...
2022-11-23T02:21:02.5391742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020330.xml
2022-11-23T02:21:02.5392229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5392409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5392798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5392992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5393015Z 
2022-11-23T02:21:02.5393106Z Running tests...
2022-11-23T02:21:02.5393376Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5393693Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5394166Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5394928Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.612s)
2022-11-23T02:21:02.5394949Z 
2022-11-23T02:21:02.5395212Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5395326Z Ran 1 test in 1.612s
2022-11-23T02:21:02.5395346Z 
2022-11-23T02:21:02.5395452Z OK (skipped=1)
2022-11-23T02:21:02.5395471Z 
2022-11-23T02:21:02.5395592Z Generating XML reports...
2022-11-23T02:21:02.5396028Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020335.xml
2022-11-23T02:21:02.5396407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5396582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5396963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5397158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5397177Z 
2022-11-23T02:21:02.5397284Z Running tests...
2022-11-23T02:21:02.5397546Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5397856Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5398174Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5398918Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.650s)
2022-11-23T02:21:02.5398943Z 
2022-11-23T02:21:02.5399256Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5399376Z Ran 1 test in 1.651s
2022-11-23T02:21:02.5399396Z 
2022-11-23T02:21:02.5399501Z OK (skipped=1)
2022-11-23T02:21:02.5399520Z 
2022-11-23T02:21:02.5399643Z Generating XML reports...
2022-11-23T02:21:02.5400098Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020339.xml
2022-11-23T02:21:02.5400463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5400639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5401024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5401214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5401233Z 
2022-11-23T02:21:02.5401322Z Running tests...
2022-11-23T02:21:02.5401586Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5401897Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5402160Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5402378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37387
2022-11-23T02:21:02.5402594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37388
2022-11-23T02:21:02.5402964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5403199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5403561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5403752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5404114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5404285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5404653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5404841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5405085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5405328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5405733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5406113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5406345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5406573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5406831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tcrsynl
2022-11-23T02:21:02.5407100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tcrsynl/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5407355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpizzbd6vp
2022-11-23T02:21:02.5407626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpizzbd6vp/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5407860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5408094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5408179Z ok (6.359s)
2022-11-23T02:21:02.5408248Z 
2022-11-23T02:21:02.5408527Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5408638Z Ran 1 test in 6.359s
2022-11-23T02:21:02.5408658Z 
2022-11-23T02:21:02.5408749Z OK
2022-11-23T02:21:02.5408767Z 
2022-11-23T02:21:02.5408887Z Generating XML reports...
2022-11-23T02:21:02.5409336Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020343.xml
2022-11-23T02:21:02.5409705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5409884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5410249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5410439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5410459Z 
2022-11-23T02:21:02.5410564Z Running tests...
2022-11-23T02:21:02.5410827Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5411138Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5411414Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5411630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37502
2022-11-23T02:21:02.5411846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37503
2022-11-23T02:21:02.5412200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5412434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5412818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5413009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5413373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5413544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5413911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5414098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5414343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5414576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5414975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5415374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5415601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5415828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5416083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpet1r45qz
2022-11-23T02:21:02.5416351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpet1r45qz/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5416602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstmu0el8
2022-11-23T02:21:02.5416869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstmu0el8/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5417085Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5417368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5417603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5417834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5417934Z ok (6.267s)
2022-11-23T02:21:02.5417954Z 
2022-11-23T02:21:02.5418224Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5418334Z Ran 1 test in 6.267s
2022-11-23T02:21:02.5418353Z 
2022-11-23T02:21:02.5418443Z OK
2022-11-23T02:21:02.5418462Z 
2022-11-23T02:21:02.5418566Z Generating XML reports...
2022-11-23T02:21:02.5419024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020352.xml
2022-11-23T02:21:02.5419394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5419568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5419947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5420135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5420154Z 
2022-11-23T02:21:02.5420260Z Running tests...
2022-11-23T02:21:02.5420520Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5420831Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5421083Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5421887Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.626s)
2022-11-23T02:21:02.5421912Z 
2022-11-23T02:21:02.5422170Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5422278Z Ran 1 test in 1.627s
2022-11-23T02:21:02.5422297Z 
2022-11-23T02:21:02.5422402Z OK (skipped=1)
2022-11-23T02:21:02.5422421Z 
2022-11-23T02:21:02.5422542Z Generating XML reports...
2022-11-23T02:21:02.5422986Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020401.xml
2022-11-23T02:21:02.5423351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5423530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5423891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5424083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5424103Z 
2022-11-23T02:21:02.5424211Z Running tests...
2022-11-23T02:21:02.5424471Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5424779Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5425066Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5425805Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.631s)
2022-11-23T02:21:02.5425829Z 
2022-11-23T02:21:02.5426088Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5426197Z Ran 1 test in 1.631s
2022-11-23T02:21:02.5426217Z 
2022-11-23T02:21:02.5426323Z OK (skipped=1)
2022-11-23T02:21:02.5426394Z 
2022-11-23T02:21:02.5426505Z Generating XML reports...
2022-11-23T02:21:02.5427268Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020405.xml
2022-11-23T02:21:02.5427662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5427836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5428213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5428407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5428427Z 
2022-11-23T02:21:02.5428533Z Running tests...
2022-11-23T02:21:02.5428795Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5429106Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5429382Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5429601Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37685
2022-11-23T02:21:02.5429817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37686
2022-11-23T02:21:02.5430184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5430357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5430732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5431022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5431391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5431550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5431920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5432104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5432348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5432592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5432992Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5433393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5433622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5433851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5434090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmh24fdb8
2022-11-23T02:21:02.5434357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmh24fdb8/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5434607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp897lcvw1
2022-11-23T02:21:02.5434872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp897lcvw1/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5435079Z <class 'torch.nn.parameter.Parameter'>
2022-11-23T02:21:02.5435184Z ok (5.761s)
2022-11-23T02:21:02.5435204Z 
2022-11-23T02:21:02.5435467Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5435577Z Ran 1 test in 5.761s
2022-11-23T02:21:02.5435596Z 
2022-11-23T02:21:02.5435669Z OK
2022-11-23T02:21:02.5435705Z 
2022-11-23T02:21:02.5435810Z Generating XML reports...
2022-11-23T02:21:02.5436323Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020409.xml
2022-11-23T02:21:02.5436709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5436884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5437260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5437449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5437473Z 
2022-11-23T02:21:02.5437580Z Running tests...
2022-11-23T02:21:02.5437839Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5438132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5438442Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5438660Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37796
2022-11-23T02:21:02.5438875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37797
2022-11-23T02:21:02.5439240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5439412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5439792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5440039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5440384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5440556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5440928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5441113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5441360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5441601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5442002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5442401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5442629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5442843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5443098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgl23owng
2022-11-23T02:21:02.5443367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgl23owng/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5443618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_v5j75a
2022-11-23T02:21:02.5443880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_v5j75a/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5443981Z ok (5.757s)
2022-11-23T02:21:02.5444000Z 
2022-11-23T02:21:02.5444270Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5444380Z Ran 1 test in 5.757s
2022-11-23T02:21:02.5444399Z 
2022-11-23T02:21:02.5444488Z OK
2022-11-23T02:21:02.5444507Z 
2022-11-23T02:21:02.5444612Z Generating XML reports...
2022-11-23T02:21:02.5445111Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020417.xml
2022-11-23T02:21:02.5445493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5445667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5446038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5446226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5446245Z 
2022-11-23T02:21:02.5446352Z Running tests...
2022-11-23T02:21:02.5446617Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5446911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5447172Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5447391Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37907
2022-11-23T02:21:02.5447605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37908
2022-11-23T02:21:02.5447969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5448141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5448516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5448702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5449151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5449306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5449674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5449863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5450108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5450351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5450750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5451142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5451379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5451606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5451842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3huck8cr
2022-11-23T02:21:02.5452115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3huck8cr/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5452368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphv2uyxof
2022-11-23T02:21:02.5452634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphv2uyxof/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5452867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5453092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5453196Z ok (6.256s)
2022-11-23T02:21:02.5453215Z 
2022-11-23T02:21:02.5453483Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5453575Z Ran 1 test in 6.256s
2022-11-23T02:21:02.5453595Z 
2022-11-23T02:21:02.5453687Z OK
2022-11-23T02:21:02.5453706Z 
2022-11-23T02:21:02.5453826Z Generating XML reports...
2022-11-23T02:21:02.5454325Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020426.xml
2022-11-23T02:21:02.5454708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5454884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5455261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5455450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5455474Z 
2022-11-23T02:21:02.5455582Z Running tests...
2022-11-23T02:21:02.5455825Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5456136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5456425Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5456640Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38022
2022-11-23T02:21:02.5456854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38023
2022-11-23T02:21:02.5457219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5457393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5457809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5458041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5458407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5458577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5458954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5459140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5459386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5459630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5460027Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5460423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5460635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5460863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5461120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm8uaev97
2022-11-23T02:21:02.5461385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm8uaev97/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5461636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzsg44_1j
2022-11-23T02:21:02.5461900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzsg44_1j/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5462730Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:21:02.5462844Z ok (6.222s)
2022-11-23T02:21:02.5462864Z 
2022-11-23T02:21:02.5463136Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5463247Z Ran 1 test in 6.222s
2022-11-23T02:21:02.5463267Z 
2022-11-23T02:21:02.5463338Z OK
2022-11-23T02:21:02.5463356Z 
2022-11-23T02:21:02.5463480Z Generating XML reports...
2022-11-23T02:21:02.5463930Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020434.xml
2022-11-23T02:21:02.5464299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5464478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5464853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5465046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5465065Z 
2022-11-23T02:21:02.5465170Z Running tests...
2022-11-23T02:21:02.5465431Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5465726Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5466007Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5466748Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.609s)
2022-11-23T02:21:02.5467071Z 
2022-11-23T02:21:02.5467370Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5467484Z Ran 1 test in 1.609s
2022-11-23T02:21:02.5467503Z 
2022-11-23T02:21:02.5467612Z OK (skipped=1)
2022-11-23T02:21:02.5467632Z 
2022-11-23T02:21:02.5467754Z Generating XML reports...
2022-11-23T02:21:02.5468204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020443.xml
2022-11-23T02:21:02.5468574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5468732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5469109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5469304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5469324Z 
2022-11-23T02:21:02.5469430Z Running tests...
2022-11-23T02:21:02.5469691Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5470005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5470263Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5470481Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38171
2022-11-23T02:21:02.5470698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38172
2022-11-23T02:21:02.5471050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5471222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5471604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5471791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5472153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5472393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5472780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5472967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5473195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5473444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5473842Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5474243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5474473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5474730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0b2qprnx
2022-11-23T02:21:02.5474998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0b2qprnx/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5475227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5475482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5fd1l8dr
2022-11-23T02:21:02.5475732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5fd1l8dr/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5476687Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:21:02.5477578Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:21:02.5478759Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T02:21:02.5478982Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-11-23T02:21:02.5480136Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1127.)
2022-11-23T02:21:02.5480371Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-11-23T02:21:02.5480610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5480845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5481779Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:21:02.5482671Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:21:02.5483553Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:21:02.5484419Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:21:02.5485293Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:21:02.5486213Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:21:02.5487082Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:21:02.5487951Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:21:02.5488819Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:21:02.5489678Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-11-23T02:21:02.5489785Z ok (4.345s)
2022-11-23T02:21:02.5489805Z 
2022-11-23T02:21:02.5490073Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5490278Z Ran 1 test in 4.346s
2022-11-23T02:21:02.5490299Z 
2022-11-23T02:21:02.5490392Z OK
2022-11-23T02:21:02.5490411Z 
2022-11-23T02:21:02.5490534Z Generating XML reports...
2022-11-23T02:21:02.5490992Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020447.xml
2022-11-23T02:21:02.5491363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5491521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5491904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5492098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5492117Z 
2022-11-23T02:21:02.5492223Z Running tests...
2022-11-23T02:21:02.5492489Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5492802Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5493053Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5493795Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.639s)
2022-11-23T02:21:02.5493816Z 
2022-11-23T02:21:02.5494075Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5494237Z Ran 1 test in 1.640s
2022-11-23T02:21:02.5494257Z 
2022-11-23T02:21:02.5494346Z OK (skipped=1)
2022-11-23T02:21:02.5494365Z 
2022-11-23T02:21:02.5494487Z Generating XML reports...
2022-11-23T02:21:02.5494943Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020454.xml
2022-11-23T02:21:02.5495314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5495488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5495863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5496054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5496073Z 
2022-11-23T02:21:02.5496180Z Running tests...
2022-11-23T02:21:02.5496424Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5496741Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5497011Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5497232Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38318
2022-11-23T02:21:02.5497450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38319
2022-11-23T02:21:02.5497817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5497990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5498371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5498560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5498909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5499079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5499448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5499690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5499942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5500185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5500586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5500981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5501216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5501428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5501687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwh11gyw
2022-11-23T02:21:02.5501959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwh11gyw/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5502214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1_19_bu
2022-11-23T02:21:02.5502476Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1_19_bu/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5503264Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T02:21:02.5503649Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T02:21:02.5504435Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1319: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-11-23T02:21:02.5504763Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-11-23T02:21:02.5504863Z ok (6.262s)
2022-11-23T02:21:02.5504883Z 
2022-11-23T02:21:02.5505130Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5505241Z Ran 1 test in 6.263s
2022-11-23T02:21:02.5505260Z 
2022-11-23T02:21:02.5505350Z OK
2022-11-23T02:21:02.5505372Z 
2022-11-23T02:21:02.5505495Z Generating XML reports...
2022-11-23T02:21:02.5505946Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020458.xml
2022-11-23T02:21:02.5506315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5506493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5507103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5507305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5507325Z 
2022-11-23T02:21:02.5507416Z Running tests...
2022-11-23T02:21:02.5507688Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5507997Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5508271Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5509091Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.653s)
2022-11-23T02:21:02.5509115Z 
2022-11-23T02:21:02.5509385Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5509496Z Ran 1 test in 1.653s
2022-11-23T02:21:02.5509515Z 
2022-11-23T02:21:02.5509620Z OK (skipped=1)
2022-11-23T02:21:02.5509639Z 
2022-11-23T02:21:02.5509761Z Generating XML reports...
2022-11-23T02:21:02.5510195Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020507.xml
2022-11-23T02:21:02.5510567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5510746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5511125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5511320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5511340Z 
2022-11-23T02:21:02.5511445Z Running tests...
2022-11-23T02:21:02.5511706Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5512017Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5512288Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5513015Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.655s)
2022-11-23T02:21:02.5513139Z 
2022-11-23T02:21:02.5513391Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5513502Z Ran 1 test in 1.655s
2022-11-23T02:21:02.5513521Z 
2022-11-23T02:21:02.5513628Z OK (skipped=1)
2022-11-23T02:21:02.5513647Z 
2022-11-23T02:21:02.5513768Z Generating XML reports...
2022-11-23T02:21:02.5514219Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020511.xml
2022-11-23T02:21:02.5514586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5514761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5515137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5515332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5515351Z 
2022-11-23T02:21:02.5515441Z Running tests...
2022-11-23T02:21:02.5515700Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5516012Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5516301Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5516518Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38531
2022-11-23T02:21:02.5516732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38532
2022-11-23T02:21:02.5517099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5517272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5517637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5517825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5518242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5518421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5518793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5518979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5519226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5519468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5519874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5520252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5520482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5520709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5520947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.5521187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.5521583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5521976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5522291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rmt8k0f
2022-11-23T02:21:02.5522561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rmt8k0f/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5522798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ydemjuw
2022-11-23T02:21:02.5523066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ydemjuw/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5523300Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5523533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5523764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5523988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5524096Z ok (6.552s)
2022-11-23T02:21:02.5524115Z 
2022-11-23T02:21:02.5524385Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5524478Z Ran 1 test in 6.552s
2022-11-23T02:21:02.5524515Z 
2022-11-23T02:21:02.5524588Z OK
2022-11-23T02:21:02.5524607Z 
2022-11-23T02:21:02.5524728Z Generating XML reports...
2022-11-23T02:21:02.5525181Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020516.xml
2022-11-23T02:21:02.5525552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5525726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5526103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5526292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5526315Z 
2022-11-23T02:21:02.5526422Z Running tests...
2022-11-23T02:21:02.5526665Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5526974Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5527295Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5527518Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38652
2022-11-23T02:21:02.5527733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38653
2022-11-23T02:21:02.5528106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5528280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5528656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5528832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5529197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5529366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5529744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5529930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5530177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5530419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5530968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5531434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5531647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5531931Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:21:02.5532154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5532423Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:21:02.5532679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1v7gzq0
2022-11-23T02:21:02.5532947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1v7gzq0/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5533199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplclvohy9
2022-11-23T02:21:02.5533471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplclvohy9/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5533705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5533921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5534149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5534380Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5534656Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:21:02.5534930Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:21:02.5535201Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:21:02.5535472Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-11-23T02:21:02.5535704Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5535988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5536211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5536440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5536710Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:21:02.5536986Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-11-23T02:21:02.5537258Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T02:21:02.5537528Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-11-23T02:21:02.5537759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5537991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5538220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5538431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5538530Z ok (7.045s)
2022-11-23T02:21:02.5538550Z 
2022-11-23T02:21:02.5538820Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5538930Z Ran 1 test in 7.045s
2022-11-23T02:21:02.5538950Z 
2022-11-23T02:21:02.5539040Z OK
2022-11-23T02:21:02.5539059Z 
2022-11-23T02:21:02.5539180Z Generating XML reports...
2022-11-23T02:21:02.5539692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020525.xml
2022-11-23T02:21:02.5540062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5540219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5540598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5540787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5540807Z 
2022-11-23T02:21:02.5540913Z Running tests...
2022-11-23T02:21:02.5541173Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5541484Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5541753Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5542500Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2022-11-23T02:21:02.5542524Z 
2022-11-23T02:21:02.5542784Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5542877Z Ran 1 test in 1.634s
2022-11-23T02:21:02.5542913Z 
2022-11-23T02:21:02.5543001Z OK (skipped=1)
2022-11-23T02:21:02.5543020Z 
2022-11-23T02:21:02.5543142Z Generating XML reports...
2022-11-23T02:21:02.5543589Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020534.xml
2022-11-23T02:21:02.5543957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5544134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5544513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5544703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5544723Z 
2022-11-23T02:21:02.5544879Z Running tests...
2022-11-23T02:21:02.5545129Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5545441Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5545713Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5545932Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38801
2022-11-23T02:21:02.5546150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38802
2022-11-23T02:21:02.5546524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5546700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5547313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5547510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5547856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5548027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5548398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5548584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5548830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5549165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5549571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5549965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5550177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5550409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5550956Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:21:02.5551508Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:21:02.5551764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45bz26w0
2022-11-23T02:21:02.5552030Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45bz26w0/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5552281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47gkb_4e
2022-11-23T02:21:02.5552545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47gkb_4e/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5552778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5553016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5553290Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T02:21:02.5553609Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-11-23T02:21:02.5553919Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T02:21:02.5554213Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-11-23T02:21:02.5554541Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T02:21:02.5554866Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-11-23T02:21:02.5555192Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T02:21:02.5555510Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-11-23T02:21:02.5555746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5555980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5556081Z ok (6.356s)
2022-11-23T02:21:02.5556102Z 
2022-11-23T02:21:02.5556360Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5556470Z Ran 1 test in 6.356s
2022-11-23T02:21:02.5556490Z 
2022-11-23T02:21:02.5556581Z OK
2022-11-23T02:21:02.5556601Z 
2022-11-23T02:21:02.5556778Z Generating XML reports...
2022-11-23T02:21:02.5557234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020538.xml
2022-11-23T02:21:02.5557644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5557826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5558207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5558401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5558421Z 
2022-11-23T02:21:02.5558511Z Running tests...
2022-11-23T02:21:02.5558770Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5559082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5559393Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5559615Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38916
2022-11-23T02:21:02.5559831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38917
2022-11-23T02:21:02.5560205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5560379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5560739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5560931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5561291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5561464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5561837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5562022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5562319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5562571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5562973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5563351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5563581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5563807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5564066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_pcb2r4
2022-11-23T02:21:02.5564332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_pcb2r4/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5564590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdg747yqa
2022-11-23T02:21:02.5564855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdg747yqa/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5565086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5565304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5565537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5565759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5565917Z ok (6.447s)
2022-11-23T02:21:02.5565937Z 
2022-11-23T02:21:02.5566204Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5566313Z Ran 1 test in 6.447s
2022-11-23T02:21:02.5566333Z 
2022-11-23T02:21:02.5566424Z OK
2022-11-23T02:21:02.5566443Z 
2022-11-23T02:21:02.5566565Z Generating XML reports...
2022-11-23T02:21:02.5567019Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020547.xml
2022-11-23T02:21:02.5567373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5567547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5567920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5568109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5568133Z 
2022-11-23T02:21:02.5568240Z Running tests...
2022-11-23T02:21:02.5568503Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5568811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5569123Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5569323Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39061
2022-11-23T02:21:02.5569539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39062
2022-11-23T02:21:02.5569908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5570085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5570456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5570647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5571009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5571232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5571619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5571787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5572030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5572273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5572672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5573071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5573300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5573530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5573786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvwy_uv_y
2022-11-23T02:21:02.5574058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvwy_uv_y/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5574294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6lwdsoz6
2022-11-23T02:21:02.5574559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6lwdsoz6/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5574790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5575097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5575325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5575558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5575662Z ok (6.467s)
2022-11-23T02:21:02.5575681Z 
2022-11-23T02:21:02.5575950Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5576042Z Ran 1 test in 6.468s
2022-11-23T02:21:02.5576082Z 
2022-11-23T02:21:02.5576155Z OK
2022-11-23T02:21:02.5576174Z 
2022-11-23T02:21:02.5576294Z Generating XML reports...
2022-11-23T02:21:02.5576744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020556.xml
2022-11-23T02:21:02.5577113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5577292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5577667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5577855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5577878Z 
2022-11-23T02:21:02.5577984Z Running tests...
2022-11-23T02:21:02.5578229Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5578539Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5578910Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5579128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39206
2022-11-23T02:21:02.5579347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39207
2022-11-23T02:21:02.5579714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5579886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5580312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5580509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5580858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5581031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5581398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5581587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5581835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5582079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5582478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5582872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5583085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5583311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5583566Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4ky5ne4
2022-11-23T02:21:02.5583833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4ky5ne4/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5584143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplsaid1b7
2022-11-23T02:21:02.5584407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplsaid1b7/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5584642Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5584875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5585105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5585321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5585420Z ok (6.425s)
2022-11-23T02:21:02.5585440Z 
2022-11-23T02:21:02.5585711Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5585820Z Ran 1 test in 6.425s
2022-11-23T02:21:02.5585843Z 
2022-11-23T02:21:02.5585933Z OK
2022-11-23T02:21:02.5585952Z 
2022-11-23T02:21:02.5586075Z Generating XML reports...
2022-11-23T02:21:02.5586526Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020605.xml
2022-11-23T02:21:02.5587127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5587294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5587681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5587868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5587889Z 
2022-11-23T02:21:02.5587996Z Running tests...
2022-11-23T02:21:02.5588258Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5588568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5588942Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5589160Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39351
2022-11-23T02:21:02.5589453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39352
2022-11-23T02:21:02.5589825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5589999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5590369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5590559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5590918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5591094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5591462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5591653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5591900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5592128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5592528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5592922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5593227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5593456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5593712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfgibrm2
2022-11-23T02:21:02.5593985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfgibrm2/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5594238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpat0iefdc
2022-11-23T02:21:02.5594507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpat0iefdc/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5594724Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5594957Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5595190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5595421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5595520Z ok (6.535s)
2022-11-23T02:21:02.5595540Z 
2022-11-23T02:21:02.5595811Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5595921Z Ran 1 test in 6.536s
2022-11-23T02:21:02.5595945Z 
2022-11-23T02:21:02.5596035Z OK
2022-11-23T02:21:02.5596053Z 
2022-11-23T02:21:02.5596158Z Generating XML reports...
2022-11-23T02:21:02.5596609Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020614.xml
2022-11-23T02:21:02.5596977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5597151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5597529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5597723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5597742Z 
2022-11-23T02:21:02.5597850Z Running tests...
2022-11-23T02:21:02.5598110Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5598455Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5598835Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5599054Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39496
2022-11-23T02:21:02.5599271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39497
2022-11-23T02:21:02.5599643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5599824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5600200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5600388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5600752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5600907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5601277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5601463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5601707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5601952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5602413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5602807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5603041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5603270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5603510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoij6qyim
2022-11-23T02:21:02.5603783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoij6qyim/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5604035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbuva_u7y
2022-11-23T02:21:02.5604298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbuva_u7y/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5604536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5604770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5605004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5605236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5605321Z ok (6.535s)
2022-11-23T02:21:02.5605357Z 
2022-11-23T02:21:02.5605611Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5605721Z Ran 1 test in 6.535s
2022-11-23T02:21:02.5605741Z 
2022-11-23T02:21:02.5605831Z OK
2022-11-23T02:21:02.5605850Z 
2022-11-23T02:21:02.5605971Z Generating XML reports...
2022-11-23T02:21:02.5606418Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020623.xml
2022-11-23T02:21:02.5606791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5606966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5607394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5607575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5607594Z 
2022-11-23T02:21:02.5607702Z Running tests...
2022-11-23T02:21:02.5607963Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5608273Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5608640Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5608861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39641
2022-11-23T02:21:02.5609077Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39642
2022-11-23T02:21:02.5609445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5609622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5609982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5610173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5610533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5610703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5611070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5611316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5611563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5611812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5612198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5612593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5612822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5613051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5613312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt721jyph
2022-11-23T02:21:02.5613579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt721jyph/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5613829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_ab8be9
2022-11-23T02:21:02.5614102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_ab8be9/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5614333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5614548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5614779Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5615007Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5615106Z ok (6.348s)
2022-11-23T02:21:02.5615128Z 
2022-11-23T02:21:02.5615396Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5615506Z Ran 1 test in 6.348s
2022-11-23T02:21:02.5615525Z 
2022-11-23T02:21:02.5615616Z OK
2022-11-23T02:21:02.5615634Z 
2022-11-23T02:21:02.5615756Z Generating XML reports...
2022-11-23T02:21:02.5616241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020632.xml
2022-11-23T02:21:02.5616625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5616800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5617180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5617370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5617389Z 
2022-11-23T02:21:02.5617495Z Running tests...
2022-11-23T02:21:02.5617759Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5618068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5618439Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5618642Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39786
2022-11-23T02:21:02.5618857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39787
2022-11-23T02:21:02.5619229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5619401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5619776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5620019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5620382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5620555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5620928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5621100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5621343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5621587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5621986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5622381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5622610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5622837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5623098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqrgqlu9
2022-11-23T02:21:02.5623369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqrgqlu9/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5623605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphk11v9t2
2022-11-23T02:21:02.5623869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphk11v9t2/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5624101Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5624341Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5624572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5624804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5624904Z ok (6.439s)
2022-11-23T02:21:02.5624977Z 
2022-11-23T02:21:02.5625256Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5625349Z Ran 1 test in 6.439s
2022-11-23T02:21:02.5625369Z 
2022-11-23T02:21:02.5625460Z OK
2022-11-23T02:21:02.5625479Z 
2022-11-23T02:21:02.5625599Z Generating XML reports...
2022-11-23T02:21:02.5626054Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020641.xml
2022-11-23T02:21:02.5626421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5626602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5627221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5627417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5627438Z 
2022-11-23T02:21:02.5627545Z Running tests...
2022-11-23T02:21:02.5627807Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5628119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5628483Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5628702Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39931
2022-11-23T02:21:02.5628918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39932
2022-11-23T02:21:02.5629446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5629620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5629996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5630169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5630536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5630709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5631077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5631263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5631512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5631755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5632154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5632552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5632765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5632992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5633245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw91v7_wr
2022-11-23T02:21:02.5633510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw91v7_wr/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5633763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0k589b_7
2022-11-23T02:21:02.5634028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0k589b_7/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5634263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5634560Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5634803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5635016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5635116Z ok (6.451s)
2022-11-23T02:21:02.5635136Z 
2022-11-23T02:21:02.5635407Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5635518Z Ran 1 test in 6.451s
2022-11-23T02:21:02.5635537Z 
2022-11-23T02:21:02.5635628Z OK
2022-11-23T02:21:02.5635652Z 
2022-11-23T02:21:02.5635778Z Generating XML reports...
2022-11-23T02:21:02.5636234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020650.xml
2022-11-23T02:21:02.5636601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5636763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5637144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5637333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5637353Z 
2022-11-23T02:21:02.5637461Z Running tests...
2022-11-23T02:21:02.5637719Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5638030Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5638468Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5638687Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40076
2022-11-23T02:21:02.5638905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40077
2022-11-23T02:21:02.5639261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5639433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5639798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5639969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5640344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5640538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5640914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5641100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5641331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5641577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5641977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5642369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5642598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5642830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5643087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplsgpyqgf
2022-11-23T02:21:02.5643457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplsgpyqgf/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5643724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppru9wp0_
2022-11-23T02:21:02.5643974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppru9wp0_/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5644207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5644439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5644670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5644905Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5645005Z ok (6.437s)
2022-11-23T02:21:02.5645026Z 
2022-11-23T02:21:02.5645296Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5645407Z Ran 1 test in 6.438s
2022-11-23T02:21:02.5645427Z 
2022-11-23T02:21:02.5645516Z OK
2022-11-23T02:21:02.5645540Z 
2022-11-23T02:21:02.5645643Z Generating XML reports...
2022-11-23T02:21:02.5646094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020659.xml
2022-11-23T02:21:02.5646462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5646638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5647017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5647262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5647282Z 
2022-11-23T02:21:02.5647388Z Running tests...
2022-11-23T02:21:02.5647653Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5647946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5648313Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5648530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40221
2022-11-23T02:21:02.5648745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40222
2022-11-23T02:21:02.5649114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5649288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5649667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5649856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5650220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5650374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5650744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5650931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5651175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5651419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5651822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5652216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5652493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5652732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5652971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptk1z3g84
2022-11-23T02:21:02.5653236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptk1z3g84/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5653486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_7v6uzb
2022-11-23T02:21:02.5653753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_7v6uzb/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5653995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5654226Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5654456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5654689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5654772Z ok (6.328s)
2022-11-23T02:21:02.5654809Z 
2022-11-23T02:21:02.5655063Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5655173Z Ran 1 test in 6.328s
2022-11-23T02:21:02.5655192Z 
2022-11-23T02:21:02.5655282Z OK
2022-11-23T02:21:02.5655301Z 
2022-11-23T02:21:02.5655422Z Generating XML reports...
2022-11-23T02:21:02.5655871Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020708.xml
2022-11-23T02:21:02.5656304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5656478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5656855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5657031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5657050Z 
2022-11-23T02:21:02.5657157Z Running tests...
2022-11-23T02:21:02.5657415Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5657764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5658075Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5658295Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40366
2022-11-23T02:21:02.5658514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40367
2022-11-23T02:21:02.5658883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5659039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5659419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5659607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5659968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5660137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5660504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5660696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5660939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5661182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5661615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5662022Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5662251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5662480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5662735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcclcbdnr
2022-11-23T02:21:02.5663009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcclcbdnr/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5663259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1c3i83br
2022-11-23T02:21:02.5663526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1c3i83br/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5663763Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5663981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5664213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5664444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5664545Z ok (6.319s)
2022-11-23T02:21:02.5664564Z 
2022-11-23T02:21:02.5664833Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5664999Z Ran 1 test in 6.320s
2022-11-23T02:21:02.5665018Z 
2022-11-23T02:21:02.5665108Z OK
2022-11-23T02:21:02.5665126Z 
2022-11-23T02:21:02.5665248Z Generating XML reports...
2022-11-23T02:21:02.5665686Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020717.xml
2022-11-23T02:21:02.5666058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5666233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5666613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5667023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5667045Z 
2022-11-23T02:21:02.5667163Z Running tests...
2022-11-23T02:21:02.5667431Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5667749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5668058Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5668260Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40511
2022-11-23T02:21:02.5668479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40512
2022-11-23T02:21:02.5668853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5669025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5669403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5669591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5669960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5670133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5670491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5670760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5671014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5671258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5671658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5672049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5672287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5672513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5672771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa1qr3th8
2022-11-23T02:21:02.5673025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa1qr3th8/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5673275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacodwf9a
2022-11-23T02:21:02.5673541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacodwf9a/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5673772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5674006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5674237Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5674538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5674640Z ok (6.456s)
2022-11-23T02:21:02.5674659Z 
2022-11-23T02:21:02.5674930Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5675027Z Ran 1 test in 6.457s
2022-11-23T02:21:02.5675046Z 
2022-11-23T02:21:02.5675136Z OK
2022-11-23T02:21:02.5675156Z 
2022-11-23T02:21:02.5675278Z Generating XML reports...
2022-11-23T02:21:02.5675728Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020726.xml
2022-11-23T02:21:02.5676096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5676269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5676645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5676838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5676857Z 
2022-11-23T02:21:02.5676947Z Running tests...
2022-11-23T02:21:02.5677212Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5677525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5677788Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5678536Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.609s)
2022-11-23T02:21:02.5678557Z 
2022-11-23T02:21:02.5678817Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5678937Z Ran 1 test in 1.609s
2022-11-23T02:21:02.5678955Z 
2022-11-23T02:21:02.5679062Z OK (skipped=1)
2022-11-23T02:21:02.5679081Z 
2022-11-23T02:21:02.5679205Z Generating XML reports...
2022-11-23T02:21:02.5679710Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020735.xml
2022-11-23T02:21:02.5680458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5680921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5681507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5681963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5682192Z 
2022-11-23T02:21:02.5682298Z Running tests...
2022-11-23T02:21:02.5682693Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5683231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5683728Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5684220Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40690
2022-11-23T02:21:02.5684668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40691
2022-11-23T02:21:02.5685280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5685713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5686289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5686760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5687393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5687843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5688416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5688885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5689319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5689811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5690459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5691172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5691684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5692154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5692650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9i2ysh5x
2022-11-23T02:21:02.5693176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9i2ysh5x/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5693717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6t6v1bj
2022-11-23T02:21:02.5694260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6t6v1bj/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5694636Z ok (6.450s)
2022-11-23T02:21:02.5694765Z 
2022-11-23T02:21:02.5695041Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5695407Z Ran 1 test in 6.450s
2022-11-23T02:21:02.5695571Z 
2022-11-23T02:21:02.5695673Z OK
2022-11-23T02:21:02.5695814Z 
2022-11-23T02:21:02.5695919Z Generating XML reports...
2022-11-23T02:21:02.5696532Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020739.xml
2022-11-23T02:21:02.5697465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5697944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5698527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5698977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5699206Z 
2022-11-23T02:21:02.5699316Z Running tests...
2022-11-23T02:21:02.5699719Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5700231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5700758Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5701261Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40801
2022-11-23T02:21:02.5701715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40802
2022-11-23T02:21:02.5702309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5702761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5703338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5703809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5704373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5704880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5705461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5705909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5706368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5707075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5707758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5708428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5708961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5709456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5709975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsemvs3gt
2022-11-23T02:21:02.5710499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsemvs3gt/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5711038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehu7afny
2022-11-23T02:21:02.5711579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehu7afny/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5712097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5712563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5713222Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpehu7afny'>
2022-11-23T02:21:02.5713687Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:21:02.5714277Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpsemvs3gt'>
2022-11-23T02:21:02.5714717Z   _warnings.warn(warn_message, ResourceWarning)
2022-11-23T02:21:02.5714998Z ok (6.158s)
2022-11-23T02:21:02.5715148Z 
2022-11-23T02:21:02.5715503Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5715821Z Ran 1 test in 6.158s
2022-11-23T02:21:02.5715982Z 
2022-11-23T02:21:02.5716081Z OK
2022-11-23T02:21:02.5716216Z 
2022-11-23T02:21:02.5716341Z Generating XML reports...
2022-11-23T02:21:02.5716936Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020748.xml
2022-11-23T02:21:02.5717651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5718121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5718724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5719212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5719421Z 
2022-11-23T02:21:02.5719530Z Running tests...
2022-11-23T02:21:02.5719933Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5720462Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5720956Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5721452Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40916
2022-11-23T02:21:02.5721903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40917
2022-11-23T02:21:02.5722505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5723034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5723610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5724080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5724643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5725090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5725659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5726168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5726624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5727140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5727822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5728521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5729031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5729542Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxl227upu
2022-11-23T02:21:02.5730105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxl227upu/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5730621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5731105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01angjz1
2022-11-23T02:21:02.5731656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01angjz1/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5732174Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5732663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5733044Z ok (4.467s)
2022-11-23T02:21:02.5733199Z 
2022-11-23T02:21:02.5733475Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5733806Z Ran 1 test in 4.467s
2022-11-23T02:21:02.5733969Z 
2022-11-23T02:21:02.5734043Z OK
2022-11-23T02:21:02.5734178Z 
2022-11-23T02:21:02.5734303Z Generating XML reports...
2022-11-23T02:21:02.5734917Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020757.xml
2022-11-23T02:21:02.5735632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5736071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5736646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5737115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5737347Z 
2022-11-23T02:21:02.5737459Z Running tests...
2022-11-23T02:21:02.5737839Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5738363Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5738876Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5739349Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41059
2022-11-23T02:21:02.5739801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41060
2022-11-23T02:21:02.5740474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5740924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5741480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5741950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5742527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5742948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5743519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5743978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5744430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5744919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5745573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5746265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5747026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5747485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5747982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3ajr24n
2022-11-23T02:21:02.5748525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3ajr24n/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5749043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvdvd50km
2022-11-23T02:21:02.5749577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvdvd50km/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5750092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5750666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5751010Z ok (6.264s)
2022-11-23T02:21:02.5751158Z 
2022-11-23T02:21:02.5751434Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5751770Z Ran 1 test in 6.265s
2022-11-23T02:21:02.5751935Z 
2022-11-23T02:21:02.5752010Z OK
2022-11-23T02:21:02.5752143Z 
2022-11-23T02:21:02.5752268Z Generating XML reports...
2022-11-23T02:21:02.5752879Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020804.xml
2022-11-23T02:21:02.5753598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5754035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5754617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5755091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5755324Z 
2022-11-23T02:21:02.5755414Z Running tests...
2022-11-23T02:21:02.5755820Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5756347Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5756884Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5757378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41174
2022-11-23T02:21:02.5757950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41175
2022-11-23T02:21:02.5758563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5759016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5759578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5760042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5760616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5761038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5761609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5762070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5762527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5763008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5763669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5764363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5764869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5765344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5765825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.5766318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.5766965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5767646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5768233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:21:02.5768738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:21:02.5769375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.5770056Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.5770608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp6gkjut
2022-11-23T02:21:02.5771158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp6gkjut/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5771673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu5e0bztm
2022-11-23T02:21:02.5772214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu5e0bztm/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5772594Z ok (5.724s)
2022-11-23T02:21:02.5772743Z 
2022-11-23T02:21:02.5772996Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5773324Z Ran 1 test in 5.724s
2022-11-23T02:21:02.5773486Z 
2022-11-23T02:21:02.5773583Z OK
2022-11-23T02:21:02.5773720Z 
2022-11-23T02:21:02.5773845Z Generating XML reports...
2022-11-23T02:21:02.5774432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020813.xml
2022-11-23T02:21:02.5775146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5775661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5776226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5776702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5776935Z 
2022-11-23T02:21:02.5777044Z Running tests...
2022-11-23T02:21:02.5777450Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5777963Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5778497Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5779007Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41297
2022-11-23T02:21:02.5779445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41298
2022-11-23T02:21:02.5780062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5780513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5781096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5781547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5782121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5782567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5783136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5783580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5784040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5784540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5785231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5785933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5786453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5787161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5787638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.5788135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.5788805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5789496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.5790016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:21:02.5790508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:21:02.5791157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.5791819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.5792368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelgm8j63
2022-11-23T02:21:02.5793008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelgm8j63/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5793543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwb9_rqm
2022-11-23T02:21:02.5794063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwb9_rqm/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5794446Z ok (15.775s)
2022-11-23T02:21:02.5794597Z 
2022-11-23T02:21:02.5794871Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5795201Z Ran 1 test in 15.775s
2022-11-23T02:21:02.5795343Z 
2022-11-23T02:21:02.5795439Z OK
2022-11-23T02:21:02.5795574Z 
2022-11-23T02:21:02.5795696Z Generating XML reports...
2022-11-23T02:21:02.5796302Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020821.xml
2022-11-23T02:21:02.5796995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5797452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5798031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5798504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5798718Z 
2022-11-23T02:21:02.5798828Z Running tests...
2022-11-23T02:21:02.5799229Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5799756Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5800293Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5800825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41420
2022-11-23T02:21:02.5801276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41421
2022-11-23T02:21:02.5801894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5802324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5802974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5803455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5804015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5804458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5805033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5805495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5805935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5806433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5807095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5807790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5808297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5808773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5809278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfj5b4tfg
2022-11-23T02:21:02.5809804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfj5b4tfg/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5810400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpst4w120e
2022-11-23T02:21:02.5810937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpst4w120e/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5811320Z ok (6.527s)
2022-11-23T02:21:02.5811470Z 
2022-11-23T02:21:02.5811734Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5812067Z Ran 1 test in 6.527s
2022-11-23T02:21:02.5812228Z 
2022-11-23T02:21:02.5812327Z OK
2022-11-23T02:21:02.5812460Z 
2022-11-23T02:21:02.5812564Z Generating XML reports...
2022-11-23T02:21:02.5813174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020839.xml
2022-11-23T02:21:02.5813891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5814344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5814904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5815373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5815604Z 
2022-11-23T02:21:02.5815709Z Running tests...
2022-11-23T02:21:02.5816121Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5816631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5817164Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5817674Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41535
2022-11-23T02:21:02.5818106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41536
2022-11-23T02:21:02.5818707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5819152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5819720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5820165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5820795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5823102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5823683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5824129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5824581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5825088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5825728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5826420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5827122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5827604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5828087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7h6pzkym
2022-11-23T02:21:02.5828628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7h6pzkym/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5829159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxqx25vb
2022-11-23T02:21:02.5829789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxqx25vb/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5830149Z ok (6.558s)
2022-11-23T02:21:02.5830295Z 
2022-11-23T02:21:02.5830571Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5830897Z Ran 1 test in 6.558s
2022-11-23T02:21:02.5831056Z 
2022-11-23T02:21:02.5831134Z OK
2022-11-23T02:21:02.5831269Z 
2022-11-23T02:21:02.5831394Z Generating XML reports...
2022-11-23T02:21:02.5832003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020848.xml
2022-11-23T02:21:02.5832716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5833149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5833726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5834200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5834427Z 
2022-11-23T02:21:02.5834516Z Running tests...
2022-11-23T02:21:02.5834918Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5835449Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5835958Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5836429Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41650
2022-11-23T02:21:02.5836873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41651
2022-11-23T02:21:02.5837475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5837904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5838484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5838947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5839523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5840012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5840595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5841052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5841502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5841979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5842633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5843328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5843833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5844309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5844803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnw6ael8b
2022-11-23T02:21:02.5845342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnw6ael8b/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5845852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8u2byz3b
2022-11-23T02:21:02.5846385Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8u2byz3b/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5846846Z ok (6.258s)
2022-11-23T02:21:02.5846993Z 
2022-11-23T02:21:02.5847265Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5847578Z Ran 1 test in 6.259s
2022-11-23T02:21:02.5847739Z 
2022-11-23T02:21:02.5847831Z OK
2022-11-23T02:21:02.5847963Z 
2022-11-23T02:21:02.5848088Z Generating XML reports...
2022-11-23T02:21:02.5848676Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020857.xml
2022-11-23T02:21:02.5849392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5849842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5850416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5850863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5851098Z 
2022-11-23T02:21:02.5851206Z Running tests...
2022-11-23T02:21:02.5851606Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5852116Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5852630Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5853117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41761
2022-11-23T02:21:02.5853567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41762
2022-11-23T02:21:02.5854156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5854600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5855177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5855631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5856200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5856638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5857316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5857814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5858267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5858767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5859409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5860101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5860622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5861093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5861580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpclmerr60
2022-11-23T02:21:02.5862120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpclmerr60/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5862643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi5dc2bgm
2022-11-23T02:21:02.5863177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi5dc2bgm/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5864216Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:21:02.5865817Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:21:02.5866975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5867460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5867810Z ok (6.228s)
2022-11-23T02:21:02.5867958Z 
2022-11-23T02:21:02.5868237Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5868544Z Ran 1 test in 6.228s
2022-11-23T02:21:02.5868706Z 
2022-11-23T02:21:02.5868798Z OK
2022-11-23T02:21:02.5868932Z 
2022-11-23T02:21:02.5869056Z Generating XML reports...
2022-11-23T02:21:02.5869672Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020906.xml
2022-11-23T02:21:02.5870371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5870822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5871397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5871864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5872077Z 
2022-11-23T02:21:02.5872185Z Running tests...
2022-11-23T02:21:02.5872585Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5873192Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5873712Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5874750Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.655s)
2022-11-23T02:21:02.5875268Z 
2022-11-23T02:21:02.5875532Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5875856Z Ran 1 test in 1.656s
2022-11-23T02:21:02.5876016Z 
2022-11-23T02:21:02.5876105Z OK (skipped=1)
2022-11-23T02:21:02.5876260Z 
2022-11-23T02:21:02.5876382Z Generating XML reports...
2022-11-23T02:21:02.5876991Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020915.xml
2022-11-23T02:21:02.5877707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5878139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5878715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5879183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5879415Z 
2022-11-23T02:21:02.5879504Z Running tests...
2022-11-23T02:21:02.5879907Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5880518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5881050Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5882066Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.611s)
2022-11-23T02:21:02.5882587Z 
2022-11-23T02:21:02.5882850Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5883173Z Ran 1 test in 1.611s
2022-11-23T02:21:02.5883333Z 
2022-11-23T02:21:02.5883439Z OK (skipped=1)
2022-11-23T02:21:02.5883591Z 
2022-11-23T02:21:02.5883695Z Generating XML reports...
2022-11-23T02:21:02.5884295Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020919.xml
2022-11-23T02:21:02.5885011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5885458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5886017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5886481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5886711Z 
2022-11-23T02:21:02.5886819Z Running tests...
2022-11-23T02:21:02.5887200Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5887730Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5888251Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5888758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41944
2022-11-23T02:21:02.5889188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41945
2022-11-23T02:21:02.5889787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5890293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5890861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5891328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5891902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5892340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5892887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5893353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5893808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5894308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5894949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5895639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5896159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5896609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5897171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hku8p2e
2022-11-23T02:21:02.5897716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hku8p2e/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5898246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0cusiam5
2022-11-23T02:21:02.5898767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0cusiam5/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5899394Z STAGE:2022-11-23 02:09:29 41944:41944 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5899962Z STAGE:2022-11-23 02:09:29 41945:41945 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5900444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5900917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5901511Z STAGE:2022-11-23 02:09:29 41944:41944 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5902101Z STAGE:2022-11-23 02:09:29 41945:41945 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5902680Z STAGE:2022-11-23 02:09:29 41945:41945 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5903288Z STAGE:2022-11-23 02:09:29 41944:41944 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5904322Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:21:02.5905881Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:21:02.5907183Z STAGE:2022-11-23 02:09:29 41944:41944 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5907772Z STAGE:2022-11-23 02:09:29 41945:41945 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5908558Z STAGE:2022-11-23 02:09:29 41945:41945 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:09:29 41944:41944 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5908931Z 
2022-11-23T02:21:02.5909281Z STAGE:2022-11-23 02:09:29 41945:41945 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5909883Z STAGE:2022-11-23 02:09:29 41944:41944 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5910238Z ok (6.753s)
2022-11-23T02:21:02.5910392Z 
2022-11-23T02:21:02.5910659Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5910966Z Ran 1 test in 6.753s
2022-11-23T02:21:02.5911127Z 
2022-11-23T02:21:02.5911221Z OK
2022-11-23T02:21:02.5911355Z 
2022-11-23T02:21:02.5911478Z Generating XML reports...
2022-11-23T02:21:02.5912069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020923.xml
2022-11-23T02:21:02.5912783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5913332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5913914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5914361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5914594Z 
2022-11-23T02:21:02.5914707Z Running tests...
2022-11-23T02:21:02.5915109Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5915617Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5916131Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5916630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42063
2022-11-23T02:21:02.5917081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42064
2022-11-23T02:21:02.5917675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5918131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5918709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5919160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5919745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5920195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5920766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5921214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5921672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5922175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5922826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5923574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5924094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5924567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5925068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrkt8n1i
2022-11-23T02:21:02.5925591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrkt8n1i/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5926117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjcpa2tti
2022-11-23T02:21:02.5926661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjcpa2tti/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5927018Z ok (5.720s)
2022-11-23T02:21:02.5927162Z 
2022-11-23T02:21:02.5927436Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5927760Z Ran 1 test in 5.721s
2022-11-23T02:21:02.5927917Z 
2022-11-23T02:21:02.5928008Z OK
2022-11-23T02:21:02.5928123Z 
2022-11-23T02:21:02.5928248Z Generating XML reports...
2022-11-23T02:21:02.5928855Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020933.xml
2022-11-23T02:21:02.5929564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5929997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5930567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5931095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5931324Z 
2022-11-23T02:21:02.5931429Z Running tests...
2022-11-23T02:21:02.5931821Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5932351Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5932871Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5933904Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.644s)
2022-11-23T02:21:02.5934418Z 
2022-11-23T02:21:02.5934660Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5934982Z Ran 1 test in 1.644s
2022-11-23T02:21:02.5935139Z 
2022-11-23T02:21:02.5935243Z OK (skipped=1)
2022-11-23T02:21:02.5935394Z 
2022-11-23T02:21:02.5935522Z Generating XML reports...
2022-11-23T02:21:02.5936111Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020941.xml
2022-11-23T02:21:02.5936826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5937273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5937827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5938290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5938523Z 
2022-11-23T02:21:02.5938629Z Running tests...
2022-11-23T02:21:02.5939033Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5939329Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5939610Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5939888Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42208
2022-11-23T02:21:02.5940110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42209
2022-11-23T02:21:02.5940490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5940663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5941044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5941230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5941597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5941752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5942131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5942317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5942563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5942804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5943203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5943651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5943940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5944172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5944418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7ng3urf
2022-11-23T02:21:02.5944688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7ng3urf/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5944945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuq4gsvjl
2022-11-23T02:21:02.5945212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuq4gsvjl/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5946130Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:21:02.5946246Z   warnings.warn(
2022-11-23T02:21:02.5947387Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:21:02.5947503Z   warnings.warn(
2022-11-23T02:21:02.5947741Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5947978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5948060Z ok (6.220s)
2022-11-23T02:21:02.5948097Z 
2022-11-23T02:21:02.5948357Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5948467Z Ran 1 test in 6.220s
2022-11-23T02:21:02.5948487Z 
2022-11-23T02:21:02.5948578Z OK
2022-11-23T02:21:02.5948596Z 
2022-11-23T02:21:02.5948716Z Generating XML reports...
2022-11-23T02:21:02.5949248Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020945.xml
2022-11-23T02:21:02.5949641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5949814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5950194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5950367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5950403Z 
2022-11-23T02:21:02.5950493Z Running tests...
2022-11-23T02:21:02.5950761Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5951070Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5951347Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5952096Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2022-11-23T02:21:02.5952118Z 
2022-11-23T02:21:02.5952376Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5952484Z Ran 1 test in 1.641s
2022-11-23T02:21:02.5952504Z 
2022-11-23T02:21:02.5952608Z OK (skipped=1)
2022-11-23T02:21:02.5952627Z 
2022-11-23T02:21:02.5952747Z Generating XML reports...
2022-11-23T02:21:02.5953272Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020954.xml
2022-11-23T02:21:02.5953644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5953821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5954207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5954396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5954416Z 
2022-11-23T02:21:02.5954521Z Running tests...
2022-11-23T02:21:02.5954784Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5955093Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5955369Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5955575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42357
2022-11-23T02:21:02.5955790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42358
2022-11-23T02:21:02.5956162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5956335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5956712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5956901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5957265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5957436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5957823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5958023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5958267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5958565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5958980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5959377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5959606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5959837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5960097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmple3vrhc1
2022-11-23T02:21:02.5960351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmple3vrhc1/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5960607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kvthzxn
2022-11-23T02:21:02.5960874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kvthzxn/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5961214Z STAGE:2022-11-23 02:10:03 42358:42358 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5961543Z STAGE:2022-11-23 02:10:03 42357:42357 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5961776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5962008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:21:02.5962346Z STAGE:2022-11-23 02:10:04 42358:42358 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5962752Z STAGE:2022-11-23 02:10:04 42358:42358 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5963067Z STAGE:2022-11-23 02:10:04 42357:42357 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5963414Z STAGE:2022-11-23 02:10:04 42357:42357 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5963742Z STAGE:2022-11-23 02:10:04 42357:42357 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.5964075Z STAGE:2022-11-23 02:10:04 42357:42357 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.5964415Z STAGE:2022-11-23 02:10:04 42357:42357 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.5964520Z ok (7.080s)
2022-11-23T02:21:02.5964540Z 
2022-11-23T02:21:02.5964800Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5964913Z Ran 1 test in 7.080s
2022-11-23T02:21:02.5964933Z 
2022-11-23T02:21:02.5965027Z OK
2022-11-23T02:21:02.5965045Z 
2022-11-23T02:21:02.5965150Z Generating XML reports...
2022-11-23T02:21:02.5965598Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020958.xml
2022-11-23T02:21:02.5965970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5966147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5966530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5966718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5966738Z 
2022-11-23T02:21:02.5966843Z Running tests...
2022-11-23T02:21:02.5967106Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5967403Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5967670Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5967892Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42476
2022-11-23T02:21:02.5968161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42477
2022-11-23T02:21:02.5968540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5968713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5969086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5969277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5969642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5969797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5970168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5970356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5970600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5970847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5971244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5971642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5971926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5972156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5972395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcf5azm47
2022-11-23T02:21:02.5972665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcf5azm47/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5972920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo08sfpru
2022-11-23T02:21:02.5973191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo08sfpru/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5973291Z ok (5.714s)
2022-11-23T02:21:02.5973311Z 
2022-11-23T02:21:02.5973576Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5973689Z Ran 1 test in 5.714s
2022-11-23T02:21:02.5973709Z 
2022-11-23T02:21:02.5973802Z OK
2022-11-23T02:21:02.5973822Z 
2022-11-23T02:21:02.5973942Z Generating XML reports...
2022-11-23T02:21:02.5974375Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021007.xml
2022-11-23T02:21:02.5974748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5974923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5975303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5975490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5975510Z 
2022-11-23T02:21:02.5975620Z Running tests...
2022-11-23T02:21:02.5975880Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5976189Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5976448Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5976664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42587
2022-11-23T02:21:02.5976882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42588
2022-11-23T02:21:02.5977301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5977484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5977860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5978051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5978409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5978586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5978936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5979124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5979370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.5979615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.5980017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5980414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.5980642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.5980924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.5981182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdz4046ez
2022-11-23T02:21:02.5981432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdz4046ez/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5981687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbil6t_6
2022-11-23T02:21:02.5981953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbil6t_6/_remote_module_non_scriptable.py
2022-11-23T02:21:02.5982054Z ok (5.753s)
2022-11-23T02:21:02.5982074Z 
2022-11-23T02:21:02.5982345Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5982454Z Ran 1 test in 5.753s
2022-11-23T02:21:02.5982474Z 
2022-11-23T02:21:02.5982567Z OK
2022-11-23T02:21:02.5982585Z 
2022-11-23T02:21:02.5982705Z Generating XML reports...
2022-11-23T02:21:02.5983140Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021016.xml
2022-11-23T02:21:02.5983515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5983688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5984072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5984260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5984279Z 
2022-11-23T02:21:02.5984385Z Running tests...
2022-11-23T02:21:02.5984649Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5984959Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5985236Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5985971Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.606s)
2022-11-23T02:21:02.5986012Z 
2022-11-23T02:21:02.5986314Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5986430Z Ran 1 test in 1.606s
2022-11-23T02:21:02.5986449Z 
2022-11-23T02:21:02.5986558Z OK (skipped=1)
2022-11-23T02:21:02.5986577Z 
2022-11-23T02:21:02.5986697Z Generating XML reports...
2022-11-23T02:21:02.5987398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021024.xml
2022-11-23T02:21:02.5987772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5987945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5988325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5988521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5988541Z 
2022-11-23T02:21:02.5988631Z Running tests...
2022-11-23T02:21:02.5988895Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5989207Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5989471Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5990213Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.612s)
2022-11-23T02:21:02.5990314Z 
2022-11-23T02:21:02.5990586Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5990696Z Ran 1 test in 1.612s
2022-11-23T02:21:02.5990715Z 
2022-11-23T02:21:02.5990820Z OK (skipped=1)
2022-11-23T02:21:02.5990839Z 
2022-11-23T02:21:02.5990960Z Generating XML reports...
2022-11-23T02:21:02.5991394Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021028.xml
2022-11-23T02:21:02.5991770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5991944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5992324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5992511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5992534Z 
2022-11-23T02:21:02.5992639Z Running tests...
2022-11-23T02:21:02.5992902Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5993213Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5993509Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5994247Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.636s)
2022-11-23T02:21:02.5994267Z 
2022-11-23T02:21:02.5994509Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5994623Z Ran 1 test in 1.636s
2022-11-23T02:21:02.5994643Z 
2022-11-23T02:21:02.5994752Z OK (skipped=1)
2022-11-23T02:21:02.5994771Z 
2022-11-23T02:21:02.5994891Z Generating XML reports...
2022-11-23T02:21:02.5995345Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021032.xml
2022-11-23T02:21:02.5995777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5995965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5996347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5996542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5996561Z 
2022-11-23T02:21:02.5996651Z Running tests...
2022-11-23T02:21:02.5996909Z ----------------------------------------------------------------------
2022-11-23T02:21:02.5997216Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.5997517Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.5997732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42800
2022-11-23T02:21:02.5997946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42801
2022-11-23T02:21:02.5998324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5998496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.5998874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.5999047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.5999408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.5999638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6000016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6000202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6000449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6000697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6001096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6001476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6001711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6001943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6002202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl98rg4tt
2022-11-23T02:21:02.6002472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl98rg4tt/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6002728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsquf6y4_
2022-11-23T02:21:02.6002994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsquf6y4_/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6003094Z ok (6.167s)
2022-11-23T02:21:02.6003114Z 
2022-11-23T02:21:02.6003384Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6003477Z Ran 1 test in 6.167s
2022-11-23T02:21:02.6003496Z 
2022-11-23T02:21:02.6003589Z OK
2022-11-23T02:21:02.6003609Z 
2022-11-23T02:21:02.6003728Z Generating XML reports...
2022-11-23T02:21:02.6004185Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021036.xml
2022-11-23T02:21:02.6004554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6004730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6005157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6005355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6005375Z 
2022-11-23T02:21:02.6005481Z Running tests...
2022-11-23T02:21:02.6005728Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6006040Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6006311Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6006535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42915
2022-11-23T02:21:02.6006750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42916
2022-11-23T02:21:02.6007122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6007298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6007678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6007849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6008212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6008381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6008760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6009006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6009250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6009497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6009901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6010298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6010511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6010887Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T02:21:02.6011145Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T02:21:02.6011375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6011750Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2022-11-23T02:21:02.6012001Z   warnings.warn("Initializing zero-element tensors is a no-op")
2022-11-23T02:21:02.6012254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfi9_b6ok
2022-11-23T02:21:02.6012521Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfi9_b6ok/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6012778Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9b9xf7rs
2022-11-23T02:21:02.6013028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9b9xf7rs/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6013127Z ok (5.844s)
2022-11-23T02:21:02.6013151Z 
2022-11-23T02:21:02.6013421Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6013530Z Ran 1 test in 5.845s
2022-11-23T02:21:02.6013549Z 
2022-11-23T02:21:02.6013642Z OK
2022-11-23T02:21:02.6013661Z 
2022-11-23T02:21:02.6013782Z Generating XML reports...
2022-11-23T02:21:02.6014282Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021045.xml
2022-11-23T02:21:02.6014669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6014826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6015204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6015395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6015416Z 
2022-11-23T02:21:02.6015523Z Running tests...
2022-11-23T02:21:02.6015784Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6016095Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6016353Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6016579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43026
2022-11-23T02:21:02.6016795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43027
2022-11-23T02:21:02.6017148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6017324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6017703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6017892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6018337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6018511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6018888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6019075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6019301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6019546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6019946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6020344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6020575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6020808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6021049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6021290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6021687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6022079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6022163Z ok (4.246s)
2022-11-23T02:21:02.6022183Z 
2022-11-23T02:21:02.6022449Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6022563Z Ran 1 test in 4.247s
2022-11-23T02:21:02.6022583Z 
2022-11-23T02:21:02.6022673Z OK
2022-11-23T02:21:02.6022693Z 
2022-11-23T02:21:02.6022814Z Generating XML reports...
2022-11-23T02:21:02.6023271Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021053.xml
2022-11-23T02:21:02.6023694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6023872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6024237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6024427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6024446Z 
2022-11-23T02:21:02.6024556Z Running tests...
2022-11-23T02:21:02.6024817Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6025137Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6025387Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6025605Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43141
2022-11-23T02:21:02.6025827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43142
2022-11-23T02:21:02.6026193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6026352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6026730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6027156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6027531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6027793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6028168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6028362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6028604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6028830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6029229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6029626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6029864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6030089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6030328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6030569Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6030971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6031358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6031461Z ok (4.244s)
2022-11-23T02:21:02.6031483Z 
2022-11-23T02:21:02.6031727Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6031838Z Ran 1 test in 4.244s
2022-11-23T02:21:02.6031861Z 
2022-11-23T02:21:02.6031951Z OK
2022-11-23T02:21:02.6031970Z 
2022-11-23T02:21:02.6032090Z Generating XML reports...
2022-11-23T02:21:02.6032546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021100.xml
2022-11-23T02:21:02.6032979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6033164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6033549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6033722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6033758Z 
2022-11-23T02:21:02.6033848Z Running tests...
2022-11-23T02:21:02.6034105Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6034415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6034698Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6035446Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.632s)
2022-11-23T02:21:02.6035466Z 
2022-11-23T02:21:02.6035730Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6035843Z Ran 1 test in 1.633s
2022-11-23T02:21:02.6035862Z 
2022-11-23T02:21:02.6035968Z OK (skipped=1)
2022-11-23T02:21:02.6035987Z 
2022-11-23T02:21:02.6036109Z Generating XML reports...
2022-11-23T02:21:02.6036538Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021107.xml
2022-11-23T02:21:02.6037010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6037187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6037564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6037761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6037780Z 
2022-11-23T02:21:02.6037887Z Running tests...
2022-11-23T02:21:02.6038149Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6038462Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6038736Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6039458Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.644s)
2022-11-23T02:21:02.6039503Z 
2022-11-23T02:21:02.6039746Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6039861Z Ran 1 test in 1.645s
2022-11-23T02:21:02.6039879Z 
2022-11-23T02:21:02.6039983Z OK (skipped=1)
2022-11-23T02:21:02.6040002Z 
2022-11-23T02:21:02.6040124Z Generating XML reports...
2022-11-23T02:21:02.6040572Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021111.xml
2022-11-23T02:21:02.6041134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6041323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6041714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6041911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6041935Z 
2022-11-23T02:21:02.6042023Z Running tests...
2022-11-23T02:21:02.6042282Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6042650Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6042928Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6043246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43324
2022-11-23T02:21:02.6043484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43325
2022-11-23T02:21:02.6043853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6044031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6044388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6044582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6044952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6045122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6045496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6045683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6045931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6046175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6046640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6047018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6047257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6047484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6047585Z ok (4.254s)
2022-11-23T02:21:02.6047605Z 
2022-11-23T02:21:02.6047867Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6047976Z Ran 1 test in 4.254s
2022-11-23T02:21:02.6047996Z 
2022-11-23T02:21:02.6048087Z OK
2022-11-23T02:21:02.6048105Z 
2022-11-23T02:21:02.6048226Z Generating XML reports...
2022-11-23T02:21:02.6048676Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021115.xml
2022-11-23T02:21:02.6049034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6049209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6049587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6049780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6049796Z 
2022-11-23T02:21:02.6049902Z Running tests...
2022-11-23T02:21:02.6050160Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6050466Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6050707Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6050906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43433
2022-11-23T02:21:02.6051127Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43434
2022-11-23T02:21:02.6051493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6051714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6052102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6052290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6052650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6052822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6053193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6053369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6053614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6053856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6054260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6054653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6054881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6055106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6055441Z STAGE:2022-11-23 02:11:26 43434:43434 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6055820Z STAGE:2022-11-23 02:11:26 43433:43433 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6056136Z STAGE:2022-11-23 02:11:26 43434:43434 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6056483Z STAGE:2022-11-23 02:11:26 43434:43434 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6056817Z STAGE:2022-11-23 02:11:26 43433:43433 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6057161Z STAGE:2022-11-23 02:11:26 43433:43433 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6057484Z STAGE:2022-11-23 02:11:26 43434:43434 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6057843Z STAGE:2022-11-23 02:11:26 43433:43433 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6058177Z STAGE:2022-11-23 02:11:26 43433:43433 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6058521Z STAGE:2022-11-23 02:11:26 43433:43433 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6058851Z STAGE:2022-11-23 02:11:26 43434:43434 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6059180Z STAGE:2022-11-23 02:11:26 43434:43434 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6059280Z ok (4.368s)
2022-11-23T02:21:02.6059300Z 
2022-11-23T02:21:02.6059561Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6059670Z Ran 1 test in 4.368s
2022-11-23T02:21:02.6059690Z 
2022-11-23T02:21:02.6059780Z OK
2022-11-23T02:21:02.6059799Z 
2022-11-23T02:21:02.6059920Z Generating XML reports...
2022-11-23T02:21:02.6060369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021122.xml
2022-11-23T02:21:02.6060734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6060896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6061275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6061515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6061535Z 
2022-11-23T02:21:02.6061647Z Running tests...
2022-11-23T02:21:02.6061909Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6062221Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6062473Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6062689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43546
2022-11-23T02:21:02.6062903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43547
2022-11-23T02:21:02.6063263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6063438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6063817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6064005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6064367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6064538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6064903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6065091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6065376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6065617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6066021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6066416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6066644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6067098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6067204Z ok (4.257s)
2022-11-23T02:21:02.6067224Z 
2022-11-23T02:21:02.6067496Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6067610Z Ran 1 test in 4.257s
2022-11-23T02:21:02.6067631Z 
2022-11-23T02:21:02.6067705Z OK
2022-11-23T02:21:02.6067724Z 
2022-11-23T02:21:02.6067872Z Generating XML reports...
2022-11-23T02:21:02.6068330Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021129.xml
2022-11-23T02:21:02.6068705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6068883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6069261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6069450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6069469Z 
2022-11-23T02:21:02.6069580Z Running tests...
2022-11-23T02:21:02.6069822Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6070143Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6070395Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T02:21:02.6070415Z 
2022-11-23T02:21:02.6070673Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6070782Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6070884Z 
2022-11-23T02:21:02.6070999Z OK (skipped=1)
2022-11-23T02:21:02.6071018Z 
2022-11-23T02:21:02.6071140Z Generating XML reports...
2022-11-23T02:21:02.6071592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021136.xml
2022-11-23T02:21:02.6071959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6072115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6072490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6072680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6072700Z 
2022-11-23T02:21:02.6072804Z Running tests...
2022-11-23T02:21:02.6073072Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6073386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6073643Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6073861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43688
2022-11-23T02:21:02.6074074Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43689
2022-11-23T02:21:02.6074424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6074597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6075062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6075250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6075613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6075786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6076155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6076341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6076567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6076812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6077218Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6077614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6077847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6078073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6078313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6078551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6078943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6079314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6079652Z STAGE:2022-11-23 02:11:42 43688:43688 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6079970Z STAGE:2022-11-23 02:11:42 43689:43689 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6080354Z STAGE:2022-11-23 02:11:42 43689:43689 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6080708Z STAGE:2022-11-23 02:11:42 43689:43689 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6081040Z STAGE:2022-11-23 02:11:42 43688:43688 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6081382Z STAGE:2022-11-23 02:11:42 43688:43688 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6081708Z STAGE:2022-11-23 02:11:42 43689:43689 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6082039Z STAGE:2022-11-23 02:11:42 43688:43688 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6082351Z STAGE:2022-11-23 02:11:42 43688:43688 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6082692Z STAGE:2022-11-23 02:11:42 43688:43688 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6083023Z STAGE:2022-11-23 02:11:42 43689:43689 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6083362Z STAGE:2022-11-23 02:11:42 43689:43689 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6083462Z ok (4.361s)
2022-11-23T02:21:02.6083482Z 
2022-11-23T02:21:02.6083746Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6083854Z Ran 1 test in 4.361s
2022-11-23T02:21:02.6083874Z 
2022-11-23T02:21:02.6083965Z OK
2022-11-23T02:21:02.6083983Z 
2022-11-23T02:21:02.6084104Z Generating XML reports...
2022-11-23T02:21:02.6084611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021138.xml
2022-11-23T02:21:02.6084980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6085155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6085535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6085725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6085744Z 
2022-11-23T02:21:02.6085852Z Running tests...
2022-11-23T02:21:02.6086110Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6086423Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6086658Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6086881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43807
2022-11-23T02:21:02.6087095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43808
2022-11-23T02:21:02.6087468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6087641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6088017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6088206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6088563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6088733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6089084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6089275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6089517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6089811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6090221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6090615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6090844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6091070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6091231Z skip: Skipped due to small world size. (4.221s)
2022-11-23T02:21:02.6091251Z 
2022-11-23T02:21:02.6091498Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6091608Z Ran 1 test in 4.221s
2022-11-23T02:21:02.6091627Z 
2022-11-23T02:21:02.6091731Z OK (skipped=1)
2022-11-23T02:21:02.6091750Z 
2022-11-23T02:21:02.6091874Z Generating XML reports...
2022-11-23T02:21:02.6092324Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021145.xml
2022-11-23T02:21:02.6092691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6092864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6093241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6093413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6093508Z 
2022-11-23T02:21:02.6093601Z Running tests...
2022-11-23T02:21:02.6093868Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6094177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6094432Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6094651Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43916
2022-11-23T02:21:02.6094864Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43917
2022-11-23T02:21:02.6095236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6095409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6095772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6095965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6096329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6096500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6096873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6097061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6097307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6097549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6097931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6098333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6098562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6098787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6098934Z ok (4.344s)
2022-11-23T02:21:02.6098955Z 
2022-11-23T02:21:02.6099227Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6099340Z Ran 1 test in 4.344s
2022-11-23T02:21:02.6099360Z 
2022-11-23T02:21:02.6099450Z OK
2022-11-23T02:21:02.6099469Z 
2022-11-23T02:21:02.6099590Z Generating XML reports...
2022-11-23T02:21:02.6100020Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021152.xml
2022-11-23T02:21:02.6100385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6100564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6100943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6101131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6101154Z 
2022-11-23T02:21:02.6101260Z Running tests...
2022-11-23T02:21:02.6101521Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6101830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6102077Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6102822Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.638s)
2022-11-23T02:21:02.6102914Z 
2022-11-23T02:21:02.6103161Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6103272Z Ran 1 test in 1.639s
2022-11-23T02:21:02.6103291Z 
2022-11-23T02:21:02.6103396Z OK (skipped=1)
2022-11-23T02:21:02.6103418Z 
2022-11-23T02:21:02.6103539Z Generating XML reports...
2022-11-23T02:21:02.6103989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021159.xml
2022-11-23T02:21:02.6104356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6104529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6104905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6105084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6105121Z 
2022-11-23T02:21:02.6105211Z Running tests...
2022-11-23T02:21:02.6105470Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6105780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6106032Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6106249Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44059
2022-11-23T02:21:02.6106464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44060
2022-11-23T02:21:02.6107018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6107198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6107566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6107760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6108120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6108372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6108758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6108945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6109189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6109431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6109810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6110211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6110438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6110670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6110907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6111147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6111544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6111932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6112101Z ok (4.327s)
2022-11-23T02:21:02.6112121Z 
2022-11-23T02:21:02.6112371Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6112482Z Ran 1 test in 4.327s
2022-11-23T02:21:02.6112501Z 
2022-11-23T02:21:02.6112592Z OK
2022-11-23T02:21:02.6112611Z 
2022-11-23T02:21:02.6112732Z Generating XML reports...
2022-11-23T02:21:02.6113183Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021203.xml
2022-11-23T02:21:02.6113551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6113725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6114099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6114287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6114310Z 
2022-11-23T02:21:02.6114400Z Running tests...
2022-11-23T02:21:02.6114660Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6114967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6115217Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6115431Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44174
2022-11-23T02:21:02.6115645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44175
2022-11-23T02:21:02.6116012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6116184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6116539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6116733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6117092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6117264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6117681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6117873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6118117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6118360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6118762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6119144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6119373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6119597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6119702Z ok (4.149s)
2022-11-23T02:21:02.6119721Z 
2022-11-23T02:21:02.6119984Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6120091Z Ran 1 test in 4.150s
2022-11-23T02:21:02.6120111Z 
2022-11-23T02:21:02.6120202Z OK
2022-11-23T02:21:02.6120221Z 
2022-11-23T02:21:02.6120342Z Generating XML reports...
2022-11-23T02:21:02.6120768Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021210.xml
2022-11-23T02:21:02.6121137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6121370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6121749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6121939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6121958Z 
2022-11-23T02:21:02.6122068Z Running tests...
2022-11-23T02:21:02.6122327Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6122636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6122877Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6123079Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44283
2022-11-23T02:21:02.6123294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44284
2022-11-23T02:21:02.6123665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6123836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6124207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6124399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6124761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6124929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6125280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6125464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6125706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6125951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6126349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6126791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6127028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6127253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6127355Z ok (4.552s)
2022-11-23T02:21:02.6127374Z 
2022-11-23T02:21:02.6127624Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6127735Z Ran 1 test in 4.552s
2022-11-23T02:21:02.6127754Z 
2022-11-23T02:21:02.6127845Z OK
2022-11-23T02:21:02.6127868Z 
2022-11-23T02:21:02.6127989Z Generating XML reports...
2022-11-23T02:21:02.6128437Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021217.xml
2022-11-23T02:21:02.6128802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6128977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6129354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6129539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6129559Z 
2022-11-23T02:21:02.6129649Z Running tests...
2022-11-23T02:21:02.6129910Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6130218Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6130541Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6130757Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44392
2022-11-23T02:21:02.6130973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44393
2022-11-23T02:21:02.6131345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6131519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6131875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6132061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6132420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6132589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6132968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6133153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6133397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6133640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6134037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6134414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6134644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6134868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6135109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6135348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6135795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6136195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6136296Z ok (4.249s)
2022-11-23T02:21:02.6136317Z 
2022-11-23T02:21:02.6136576Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6136672Z Ran 1 test in 4.249s
2022-11-23T02:21:02.6136691Z 
2022-11-23T02:21:02.6136782Z OK
2022-11-23T02:21:02.6136801Z 
2022-11-23T02:21:02.6136922Z Generating XML reports...
2022-11-23T02:21:02.6137373Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021224.xml
2022-11-23T02:21:02.6137740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6137914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6138293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6138483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6138502Z 
2022-11-23T02:21:02.6138591Z Running tests...
2022-11-23T02:21:02.6138850Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6139156Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6139417Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6139692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44507
2022-11-23T02:21:02.6139906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44508
2022-11-23T02:21:02.6140276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6140452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6140828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6141001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6141360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6141530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6141903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6142094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6142338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6142585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6142984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6143360Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6143591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6143814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6144059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6144298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6144694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6145128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6145236Z ok (4.277s)
2022-11-23T02:21:02.6145255Z 
2022-11-23T02:21:02.6145519Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6145614Z Ran 1 test in 4.277s
2022-11-23T02:21:02.6145651Z 
2022-11-23T02:21:02.6145725Z OK
2022-11-23T02:21:02.6145743Z 
2022-11-23T02:21:02.6145862Z Generating XML reports...
2022-11-23T02:21:02.6146307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021231.xml
2022-11-23T02:21:02.6146678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6147024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6147425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6147614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6147634Z 
2022-11-23T02:21:02.6147743Z Running tests...
2022-11-23T02:21:02.6147986Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6148296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6148557Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6148774Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44622
2022-11-23T02:21:02.6149094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44623
2022-11-23T02:21:02.6149467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6149640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6150019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6150191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6150554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6150724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6151094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6151285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6151528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6151771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6152171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6152562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6152774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6153000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6153255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_cqa1w6
2022-11-23T02:21:02.6153527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_cqa1w6/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6153782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6vlvo7s
2022-11-23T02:21:02.6154048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6vlvo7s/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6154209Z ok (6.164s)
2022-11-23T02:21:02.6154231Z 
2022-11-23T02:21:02.6154502Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6154595Z Ran 1 test in 6.164s
2022-11-23T02:21:02.6154631Z 
2022-11-23T02:21:02.6154706Z OK
2022-11-23T02:21:02.6154725Z 
2022-11-23T02:21:02.6154846Z Generating XML reports...
2022-11-23T02:21:02.6155291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021237.xml
2022-11-23T02:21:02.6155660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6155839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6156214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6156404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6156427Z 
2022-11-23T02:21:02.6156533Z Running tests...
2022-11-23T02:21:02.6156774Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6157082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6157318Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6157534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44737
2022-11-23T02:21:02.6157748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44738
2022-11-23T02:21:02.6158226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6158398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6158775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6158950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6159316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6159487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6159857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6160042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6160289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6160535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6160935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6161333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6161546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6161774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6161873Z ok (4.165s)
2022-11-23T02:21:02.6161892Z 
2022-11-23T02:21:02.6162158Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6162267Z Ran 1 test in 4.166s
2022-11-23T02:21:02.6162286Z 
2022-11-23T02:21:02.6162380Z OK
2022-11-23T02:21:02.6162399Z 
2022-11-23T02:21:02.6162518Z Generating XML reports...
2022-11-23T02:21:02.6162989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021246.xml
2022-11-23T02:21:02.6163356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6163566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6163955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6164144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6164163Z 
2022-11-23T02:21:02.6164270Z Running tests...
2022-11-23T02:21:02.6164529Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6164837Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6165077Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6165296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44846
2022-11-23T02:21:02.6165495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44847
2022-11-23T02:21:02.6165866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6166038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6166414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6166603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6166961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6167186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6167557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6167743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6167973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6168216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6168613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6169006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6169235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6169465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6169563Z ok (4.215s)
2022-11-23T02:21:02.6169582Z 
2022-11-23T02:21:02.6169843Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6169937Z Ran 1 test in 4.215s
2022-11-23T02:21:02.6169975Z 
2022-11-23T02:21:02.6170049Z OK
2022-11-23T02:21:02.6170067Z 
2022-11-23T02:21:02.6170191Z Generating XML reports...
2022-11-23T02:21:02.6170636Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021253.xml
2022-11-23T02:21:02.6171001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6171175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6171551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6171744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6171763Z 
2022-11-23T02:21:02.6171870Z Running tests...
2022-11-23T02:21:02.6172117Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6172428Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6172746Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6172969Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44955
2022-11-23T02:21:02.6173183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44956
2022-11-23T02:21:02.6173552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6173724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6174099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6174274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6174637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6174811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6175185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6175370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6175617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6175900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6176482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6196185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6196562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6196816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6197220Z STAGE:2022-11-23 02:13:03 44955:44955 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6197559Z STAGE:2022-11-23 02:13:03 44956:44956 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6197904Z STAGE:2022-11-23 02:13:03 44955:44955 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6198266Z STAGE:2022-11-23 02:13:03 44955:44955 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6198790Z STAGE:2022-11-23 02:13:03 44956:44956 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6199154Z STAGE:2022-11-23 02:13:03 44956:44956 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6199267Z ok (4.355s)
2022-11-23T02:21:02.6199290Z 
2022-11-23T02:21:02.6199567Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6199688Z Ran 1 test in 4.355s
2022-11-23T02:21:02.6199709Z 
2022-11-23T02:21:02.6199806Z OK
2022-11-23T02:21:02.6199826Z 
2022-11-23T02:21:02.6199956Z Generating XML reports...
2022-11-23T02:21:02.6200427Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021300.xml
2022-11-23T02:21:02.6200801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6200984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6201421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6201631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6201656Z 
2022-11-23T02:21:02.6201773Z Running tests...
2022-11-23T02:21:02.6202047Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6202505Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6202791Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6203017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45068
2022-11-23T02:21:02.6203240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45069
2022-11-23T02:21:02.6203704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6203892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6204328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6204527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6204901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6205078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6205461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6205656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6205908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6206138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6206633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6207034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6207270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6207504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6207843Z STAGE:2022-11-23 02:13:10 45068:45068 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6208170Z STAGE:2022-11-23 02:13:10 45069:45069 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6208504Z STAGE:2022-11-23 02:13:10 45068:45068 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6208854Z STAGE:2022-11-23 02:13:10 45068:45068 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6209177Z STAGE:2022-11-23 02:13:10 45069:45069 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6209526Z STAGE:2022-11-23 02:13:10 45069:45069 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6209631Z ok (4.257s)
2022-11-23T02:21:02.6209655Z 
2022-11-23T02:21:02.6209924Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6210038Z Ran 1 test in 4.257s
2022-11-23T02:21:02.6210058Z 
2022-11-23T02:21:02.6210154Z OK
2022-11-23T02:21:02.6210173Z 
2022-11-23T02:21:02.6210301Z Generating XML reports...
2022-11-23T02:21:02.6210757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021306.xml
2022-11-23T02:21:02.6211112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6211300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6211685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6211881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6211901Z 
2022-11-23T02:21:02.6212064Z Running tests...
2022-11-23T02:21:02.6212341Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6212660Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6212951Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6213177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45181
2022-11-23T02:21:02.6213379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45182
2022-11-23T02:21:02.6213761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6213940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6214326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6214526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6214893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6215070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6215445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6215618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6215871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6216205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6216612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6217014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6217247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6217494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6217727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6217972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6218359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6218760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6219008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:21:02.6219251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:21:02.6219753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.6220309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.6220629Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:21:02.6227347Z ok (22.385s)
2022-11-23T02:21:02.6227386Z 
2022-11-23T02:21:02.6227716Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6227818Z Ran 1 test in 22.385s
2022-11-23T02:21:02.6227854Z 
2022-11-23T02:21:02.6227927Z OK
2022-11-23T02:21:02.6227946Z 
2022-11-23T02:21:02.6228070Z Generating XML reports...
2022-11-23T02:21:02.6228632Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021313.xml
2022-11-23T02:21:02.6229023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6229197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6229574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6229767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6229787Z 
2022-11-23T02:21:02.6229894Z Running tests...
2022-11-23T02:21:02.6230142Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6230455Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6230752Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6230974Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45308
2022-11-23T02:21:02.6231188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45309
2022-11-23T02:21:02.6231552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6231721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6232086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6232354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6232723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6232886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6233247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6233429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6233672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6233908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6234299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6234693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6234922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6235150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6235375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6235612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6235999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6236381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6236608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:21:02.6236840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:21:02.6237222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.6237654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.6237885Z [E ProcessGroupGloo.cpp:2802] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:21:02.6238104Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:21:02.6238197Z ok (22.389s)
2022-11-23T02:21:02.6238217Z 
2022-11-23T02:21:02.6238476Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6238581Z Ran 1 test in 22.389s
2022-11-23T02:21:02.6238601Z 
2022-11-23T02:21:02.6238684Z OK
2022-11-23T02:21:02.6238703Z 
2022-11-23T02:21:02.6238822Z Generating XML reports...
2022-11-23T02:21:02.6239258Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021338.xml
2022-11-23T02:21:02.6260466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6260657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6261046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6261229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6261249Z 
2022-11-23T02:21:02.6261347Z Running tests...
2022-11-23T02:21:02.6261602Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6261904Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6262166Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6262465Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45435
2022-11-23T02:21:02.6262672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45436
2022-11-23T02:21:02.6263041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6263206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6263573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6263754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6264108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6264271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6264630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6264808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6265043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6265280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6265670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6266055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6266275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6266492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6266645Z skip: Skipped due to small world size. (4.146s)
2022-11-23T02:21:02.6266665Z 
2022-11-23T02:21:02.6267148Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6267259Z Ran 1 test in 4.146s
2022-11-23T02:21:02.6267279Z 
2022-11-23T02:21:02.6267377Z OK (skipped=1)
2022-11-23T02:21:02.6267395Z 
2022-11-23T02:21:02.6267594Z Generating XML reports...
2022-11-23T02:21:02.6268060Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021403.xml
2022-11-23T02:21:02.6268420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6268585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6268952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6269129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6269158Z 
2022-11-23T02:21:02.6269248Z Running tests...
2022-11-23T02:21:02.6269502Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6269804Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6270140Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6270451Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45544
2022-11-23T02:21:02.6270663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45545
2022-11-23T02:21:02.6271028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6271195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6271555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6271825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6272183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6272349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6272714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6272892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6273129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6273363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6273744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6274137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6274357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6274580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6274899Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-11-23T02:21:02.6274995Z ok (6.329s)
2022-11-23T02:21:02.6275015Z 
2022-11-23T02:21:02.6275275Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6275377Z Ran 1 test in 6.329s
2022-11-23T02:21:02.6275397Z 
2022-11-23T02:21:02.6275480Z OK
2022-11-23T02:21:02.6275499Z 
2022-11-23T02:21:02.6275604Z Generating XML reports...
2022-11-23T02:21:02.6276048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021410.xml
2022-11-23T02:21:02.6276412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6276577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6277001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6277187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6277207Z 
2022-11-23T02:21:02.6277305Z Running tests...
2022-11-23T02:21:02.6277554Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6277860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6278145Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6278369Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45653
2022-11-23T02:21:02.6278582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45654
2022-11-23T02:21:02.6278953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6279131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6279490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6279678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6280039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6280210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6280580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6280824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6281069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6281315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6281701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6282094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6282324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6282549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6282786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6283031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6283429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6283826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6284042Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-11-23T02:21:02.6284199Z No ranks successfully processed in monitoredBarrier.
2022-11-23T02:21:02.6284424Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-11-23T02:21:02.6284528Z ok (4.355s)
2022-11-23T02:21:02.6284547Z 
2022-11-23T02:21:02.6284813Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6284925Z Ran 1 test in 4.355s
2022-11-23T02:21:02.6284948Z 
2022-11-23T02:21:02.6285040Z OK
2022-11-23T02:21:02.6285060Z 
2022-11-23T02:21:02.6285182Z Generating XML reports...
2022-11-23T02:21:02.6285635Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021419.xml
2022-11-23T02:21:02.6286055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6286217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6286600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6286787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6286807Z 
2022-11-23T02:21:02.6286915Z Running tests...
2022-11-23T02:21:02.6287173Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6287484Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6287767Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6287982Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45768
2022-11-23T02:21:02.6288182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45769
2022-11-23T02:21:02.6288551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6288724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6289100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6289290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6289648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6289880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6290256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6290442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6290673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6290918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6291313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6291708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6291938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6292169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6292409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6292656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6293051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6293424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6293656Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-11-23T02:21:02.6293757Z ok (4.584s)
2022-11-23T02:21:02.6293777Z 
2022-11-23T02:21:02.6294041Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6294157Z Ran 1 test in 4.584s
2022-11-23T02:21:02.6294177Z 
2022-11-23T02:21:02.6294271Z OK
2022-11-23T02:21:02.6294289Z 
2022-11-23T02:21:02.6294414Z Generating XML reports...
2022-11-23T02:21:02.6294863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021426.xml
2022-11-23T02:21:02.6295266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6295448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6295830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6296020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6296039Z 
2022-11-23T02:21:02.6296147Z Running tests...
2022-11-23T02:21:02.6296407Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6296724Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6297001Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6297220Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45883
2022-11-23T02:21:02.6297422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45884
2022-11-23T02:21:02.6297791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6297965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6298344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6298533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6298899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6299130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6299507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6299681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6299926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6300167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6300564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6300957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6301187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6301416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6301573Z skip: Skipped due to small world size. (4.146s)
2022-11-23T02:21:02.6301592Z 
2022-11-23T02:21:02.6301857Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6301950Z Ran 1 test in 4.147s
2022-11-23T02:21:02.6301969Z 
2022-11-23T02:21:02.6302078Z OK (skipped=1)
2022-11-23T02:21:02.6302096Z 
2022-11-23T02:21:02.6302217Z Generating XML reports...
2022-11-23T02:21:02.6302666Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021433.xml
2022-11-23T02:21:02.6303035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6303210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6303593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6303783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6303802Z 
2022-11-23T02:21:02.6303909Z Running tests...
2022-11-23T02:21:02.6304202Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6304526Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6304926Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T02:21:02.6304947Z 
2022-11-23T02:21:02.6305204Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6305315Z Ran 1 test in 0.003s
2022-11-23T02:21:02.6305335Z 
2022-11-23T02:21:02.6305442Z OK (skipped=1)
2022-11-23T02:21:02.6305462Z 
2022-11-23T02:21:02.6305583Z Generating XML reports...
2022-11-23T02:21:02.6306034Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021439.xml
2022-11-23T02:21:02.6306405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6306566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6307133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6307331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6307351Z 
2022-11-23T02:21:02.6307460Z Running tests...
2022-11-23T02:21:02.6307731Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6308041Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6308442Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T02:21:02.6308558Z 
2022-11-23T02:21:02.6308835Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6308929Z Ran 1 test in 0.003s
2022-11-23T02:21:02.6308967Z 
2022-11-23T02:21:02.6309055Z OK (skipped=1)
2022-11-23T02:21:02.6309074Z 
2022-11-23T02:21:02.6309202Z Generating XML reports...
2022-11-23T02:21:02.6309647Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021442.xml
2022-11-23T02:21:02.6310012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6310187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6310558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6310747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6310771Z 
2022-11-23T02:21:02.6310878Z Running tests...
2022-11-23T02:21:02.6311122Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6311432Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6311833Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-11-23T02:21:02.6311852Z 
2022-11-23T02:21:02.6312111Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6312220Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6312239Z 
2022-11-23T02:21:02.6312345Z OK (skipped=1)
2022-11-23T02:21:02.6312364Z 
2022-11-23T02:21:02.6312486Z Generating XML reports...
2022-11-23T02:21:02.6312925Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021444.xml
2022-11-23T02:21:02.6313298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6313453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6313825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6314081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6314103Z 
2022-11-23T02:21:02.6314214Z Running tests...
2022-11-23T02:21:02.6314477Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6314786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6315181Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-11-23T02:21:02.6315200Z 
2022-11-23T02:21:02.6315457Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6315570Z Ran 1 test in 0.003s
2022-11-23T02:21:02.6315589Z 
2022-11-23T02:21:02.6315678Z OK (skipped=1)
2022-11-23T02:21:02.6315696Z 
2022-11-23T02:21:02.6315818Z Generating XML reports...
2022-11-23T02:21:02.6316265Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021447.xml
2022-11-23T02:21:02.6316635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6316808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6317185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6317373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6317392Z 
2022-11-23T02:21:02.6317500Z Running tests...
2022-11-23T02:21:02.6317741Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6318109Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6318410Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-11-23T02:21:02.6318430Z 
2022-11-23T02:21:02.6318689Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6318800Z Ran 1 test in 0.003s
2022-11-23T02:21:02.6318819Z 
2022-11-23T02:21:02.6318925Z OK (skipped=1)
2022-11-23T02:21:02.6318943Z 
2022-11-23T02:21:02.6319066Z Generating XML reports...
2022-11-23T02:21:02.6319510Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021449.xml
2022-11-23T02:21:02.6319877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6320033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6320415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6320604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6320623Z 
2022-11-23T02:21:02.6320732Z Running tests...
2022-11-23T02:21:02.6320996Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6321307Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6321556Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:21:02.6321575Z 
2022-11-23T02:21:02.6321833Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6321943Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6321962Z 
2022-11-23T02:21:02.6322050Z OK (skipped=1)
2022-11-23T02:21:02.6322068Z 
2022-11-23T02:21:02.6322194Z Generating XML reports...
2022-11-23T02:21:02.6322640Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021451.xml
2022-11-23T02:21:02.6323009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6323235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6323624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6323813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6323833Z 
2022-11-23T02:21:02.6323940Z Running tests...
2022-11-23T02:21:02.6324199Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6324490Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6324761Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:21:02.6324784Z 
2022-11-23T02:21:02.6325040Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6325150Z Ran 1 test in 0.003s
2022-11-23T02:21:02.6325170Z 
2022-11-23T02:21:02.6325278Z OK (skipped=1)
2022-11-23T02:21:02.6325297Z 
2022-11-23T02:21:02.6325423Z Generating XML reports...
2022-11-23T02:21:02.6325866Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021454.xml
2022-11-23T02:21:02.6326236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6326392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6326770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6326959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6327032Z 
2022-11-23T02:21:02.6327145Z Running tests...
2022-11-23T02:21:02.6327409Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6327718Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6328028Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:21:02.6328048Z 
2022-11-23T02:21:02.6328306Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6328417Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6328436Z 
2022-11-23T02:21:02.6328525Z OK (skipped=1)
2022-11-23T02:21:02.6328562Z 
2022-11-23T02:21:02.6328666Z Generating XML reports...
2022-11-23T02:21:02.6329108Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021456.xml
2022-11-23T02:21:02.6329484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6329657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6330030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6330221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6330241Z 
2022-11-23T02:21:02.6330348Z Running tests...
2022-11-23T02:21:02.6330607Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6330899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6331199Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6331415Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46256
2022-11-23T02:21:02.6331636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46257
2022-11-23T02:21:02.6332004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6332177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6332608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6332805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6333172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6333326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6333696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6333884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6334130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6334376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6334780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6335179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6335406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6335614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6335717Z ok (4.272s)
2022-11-23T02:21:02.6335736Z 
2022-11-23T02:21:02.6335997Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6336177Z Ran 1 test in 4.272s
2022-11-23T02:21:02.6336196Z 
2022-11-23T02:21:02.6336288Z OK
2022-11-23T02:21:02.6336308Z 
2022-11-23T02:21:02.6336432Z Generating XML reports...
2022-11-23T02:21:02.6336888Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021458.xml
2022-11-23T02:21:02.6337255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6337433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6337792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6337982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6338001Z 
2022-11-23T02:21:02.6338110Z Running tests...
2022-11-23T02:21:02.6338371Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6338686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6338981Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6339202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46365
2022-11-23T02:21:02.6339417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46366
2022-11-23T02:21:02.6339769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6339943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6340319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6340506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6340873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6341043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6341412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6341658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6341911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6342136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6342538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6342932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6343166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6343393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6343494Z ok (4.262s)
2022-11-23T02:21:02.6343514Z 
2022-11-23T02:21:02.6343778Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6343888Z Ran 1 test in 4.262s
2022-11-23T02:21:02.6343907Z 
2022-11-23T02:21:02.6343981Z OK
2022-11-23T02:21:02.6344018Z 
2022-11-23T02:21:02.6344121Z Generating XML reports...
2022-11-23T02:21:02.6344568Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021505.xml
2022-11-23T02:21:02.6344940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6345112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6345546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6345735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6345754Z 
2022-11-23T02:21:02.6345861Z Running tests...
2022-11-23T02:21:02.6346125Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6346417Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6346692Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-11-23T02:21:02.6346711Z 
2022-11-23T02:21:02.6347212Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6347325Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6347347Z 
2022-11-23T02:21:02.6347455Z OK (skipped=1)
2022-11-23T02:21:02.6347475Z 
2022-11-23T02:21:02.6347602Z Generating XML reports...
2022-11-23T02:21:02.6348058Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021512.xml
2022-11-23T02:21:02.6348426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6348602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6348961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6349149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6349168Z 
2022-11-23T02:21:02.6349276Z Running tests...
2022-11-23T02:21:02.6349534Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6349843Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6350140Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.006s)
2022-11-23T02:21:02.6350164Z 
2022-11-23T02:21:02.6350421Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6350532Z Ran 1 test in 0.006s
2022-11-23T02:21:02.6350551Z 
2022-11-23T02:21:02.6350657Z OK (skipped=1)
2022-11-23T02:21:02.6350676Z 
2022-11-23T02:21:02.6350856Z Generating XML reports...
2022-11-23T02:21:02.6351321Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021515.xml
2022-11-23T02:21:02.6351688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6351863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6352240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6352435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6352454Z 
2022-11-23T02:21:02.6352562Z Running tests...
2022-11-23T02:21:02.6352824Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6353117Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6353400Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6354154Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.670s)
2022-11-23T02:21:02.6354175Z 
2022-11-23T02:21:02.6354434Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6354545Z Ran 1 test in 1.670s
2022-11-23T02:21:02.6354686Z 
2022-11-23T02:21:02.6354799Z OK (skipped=1)
2022-11-23T02:21:02.6354818Z 
2022-11-23T02:21:02.6354941Z Generating XML reports...
2022-11-23T02:21:02.6355390Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021517.xml
2022-11-23T02:21:02.6355763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6355936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6356293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6356484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6356503Z 
2022-11-23T02:21:02.6356611Z Running tests...
2022-11-23T02:21:02.6356874Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6357230Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6357509Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6357730Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46574
2022-11-23T02:21:02.6357949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46575
2022-11-23T02:21:02.6358320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6358477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6358854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6359043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6359408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6359584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6359956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6360198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6360451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6360676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6361080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6361476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6361705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6361935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6362189Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6ymqkfb
2022-11-23T02:21:02.6362446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmkbmxt0p
2022-11-23T02:21:02.6362715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6ymqkfb/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6362979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmkbmxt0p/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6363062Z ok (6.254s)
2022-11-23T02:21:02.6363081Z 
2022-11-23T02:21:02.6363347Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6363459Z Ran 1 test in 6.254s
2022-11-23T02:21:02.6363478Z 
2022-11-23T02:21:02.6363572Z OK
2022-11-23T02:21:02.6363591Z 
2022-11-23T02:21:02.6363768Z Generating XML reports...
2022-11-23T02:21:02.6364219Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021521.xml
2022-11-23T02:21:02.6364588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6364768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6365128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6365319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6365339Z 
2022-11-23T02:21:02.6365449Z Running tests...
2022-11-23T02:21:02.6365709Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6366019Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6366290Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6366514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46689
2022-11-23T02:21:02.6366731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46690
2022-11-23T02:21:02.6367104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6367260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6367640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6367828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6368188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6368361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6368736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6368921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6369166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6369444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6369852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6370248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6370475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6370701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6370806Z ok (6.938s)
2022-11-23T02:21:02.6370826Z 
2022-11-23T02:21:02.6371088Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6371199Z Ran 1 test in 6.938s
2022-11-23T02:21:02.6371218Z 
2022-11-23T02:21:02.6371310Z OK
2022-11-23T02:21:02.6371329Z 
2022-11-23T02:21:02.6371437Z Generating XML reports...
2022-11-23T02:21:02.6371888Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021530.xml
2022-11-23T02:21:02.6372258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6372434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6372811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6373002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6373086Z 
2022-11-23T02:21:02.6373199Z Running tests...
2022-11-23T02:21:02.6373462Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6373773Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6374046Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6374262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46801
2022-11-23T02:21:02.6374478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46802
2022-11-23T02:21:02.6374847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6375022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6375396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6375590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6375949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6376102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6376476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6376664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6376908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6377153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6377552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6377951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6378181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6378461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6378550Z ok (6.855s)
2022-11-23T02:21:02.6378569Z 
2022-11-23T02:21:02.6378838Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6378948Z Ran 1 test in 6.855s
2022-11-23T02:21:02.6378968Z 
2022-11-23T02:21:02.6379061Z OK
2022-11-23T02:21:02.6379080Z 
2022-11-23T02:21:02.6379205Z Generating XML reports...
2022-11-23T02:21:02.6379653Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021539.xml
2022-11-23T02:21:02.6380023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6380201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6380581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6380757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6380776Z 
2022-11-23T02:21:02.6380885Z Running tests...
2022-11-23T02:21:02.6381143Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6381451Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6381728Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6382472Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.644s)
2022-11-23T02:21:02.6382544Z 
2022-11-23T02:21:02.6382814Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6382926Z Ran 1 test in 1.644s
2022-11-23T02:21:02.6382945Z 
2022-11-23T02:21:02.6383055Z OK (skipped=1)
2022-11-23T02:21:02.6383075Z 
2022-11-23T02:21:02.6383181Z Generating XML reports...
2022-11-23T02:21:02.6383629Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021549.xml
2022-11-23T02:21:02.6383997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6384171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6384547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6384742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6384760Z 
2022-11-23T02:21:02.6384868Z Running tests...
2022-11-23T02:21:02.6385132Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6385448Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6385726Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6386466Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.623s)
2022-11-23T02:21:02.6386486Z 
2022-11-23T02:21:02.6386743Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6387130Z Ran 1 test in 1.623s
2022-11-23T02:21:02.6387151Z 
2022-11-23T02:21:02.6387260Z OK (skipped=1)
2022-11-23T02:21:02.6387279Z 
2022-11-23T02:21:02.6387403Z Generating XML reports...
2022-11-23T02:21:02.6387860Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021553.xml
2022-11-23T02:21:02.6388311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6388495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6388883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6389054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6389073Z 
2022-11-23T02:21:02.6389180Z Running tests...
2022-11-23T02:21:02.6389440Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6389755Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6390064Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6390285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46981
2022-11-23T02:21:02.6390503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46982
2022-11-23T02:21:02.6390872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6391027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6391405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6391593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6392038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6392210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6392579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6392770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6393016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6393263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6393647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6394041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6394275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6394500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6394647Z skip: Need at least 4 CUDA devices (4.114s)
2022-11-23T02:21:02.6394667Z 
2022-11-23T02:21:02.6394933Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6395044Z Ran 1 test in 4.115s
2022-11-23T02:21:02.6395063Z 
2022-11-23T02:21:02.6395170Z OK (skipped=1)
2022-11-23T02:21:02.6395188Z 
2022-11-23T02:21:02.6395292Z Generating XML reports...
2022-11-23T02:21:02.6395743Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021557.xml
2022-11-23T02:21:02.6396112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6396288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6396668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6396859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6396878Z 
2022-11-23T02:21:02.6396986Z Running tests...
2022-11-23T02:21:02.6397296Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6397619Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6397928Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6398146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47090
2022-11-23T02:21:02.6398361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47091
2022-11-23T02:21:02.6398733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6398908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6399286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6399480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6399839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6400010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6400364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6400552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6400796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6401098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6401502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6401900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6402131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6402359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6402488Z skip: Need at least 4 CUDA devices (4.132s)
2022-11-23T02:21:02.6402526Z 
2022-11-23T02:21:02.6402772Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6402883Z Ran 1 test in 4.132s
2022-11-23T02:21:02.6402902Z 
2022-11-23T02:21:02.6403013Z OK (skipped=1)
2022-11-23T02:21:02.6403032Z 
2022-11-23T02:21:02.6403154Z Generating XML reports...
2022-11-23T02:21:02.6403605Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021604.xml
2022-11-23T02:21:02.6403975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6404149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6404528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6404698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6404736Z 
2022-11-23T02:21:02.6404826Z Running tests...
2022-11-23T02:21:02.6405086Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6405400Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6405688Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6406480Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.648s)
2022-11-23T02:21:02.6406502Z 
2022-11-23T02:21:02.6406769Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6406879Z Ran 1 test in 1.649s
2022-11-23T02:21:02.6406899Z 
2022-11-23T02:21:02.6407005Z OK (skipped=1)
2022-11-23T02:21:02.6407024Z 
2022-11-23T02:21:02.6407147Z Generating XML reports...
2022-11-23T02:21:02.6407576Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021610.xml
2022-11-23T02:21:02.6407947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6408126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6408504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6408700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6408720Z 
2022-11-23T02:21:02.6408827Z Running tests...
2022-11-23T02:21:02.6409088Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6409399Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6409645Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6409863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47233
2022-11-23T02:21:02.6410138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47234
2022-11-23T02:21:02.6410511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6410685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6411065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6411256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6411617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6411789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6412142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6412328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6412578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6412823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6413226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6413622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6413851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6414077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6414317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6414539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6414941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6415331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6415713Z STAGE:2022-11-23 02:16:18 47233:47233 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6416047Z STAGE:2022-11-23 02:16:18 47234:47234 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6416380Z STAGE:2022-11-23 02:16:18 47234:47234 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6416705Z STAGE:2022-11-23 02:16:18 47233:47233 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6417052Z STAGE:2022-11-23 02:16:18 47234:47234 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6417403Z STAGE:2022-11-23 02:16:18 47233:47233 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6417711Z STAGE:2022-11-23 02:16:18 47234:47234 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6418035Z STAGE:2022-11-23 02:16:18 47233:47233 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6418366Z STAGE:2022-11-23 02:16:18 47233:47233 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6418709Z STAGE:2022-11-23 02:16:18 47233:47233 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6419035Z STAGE:2022-11-23 02:16:18 47234:47234 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6419377Z STAGE:2022-11-23 02:16:18 47234:47234 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6419479Z ok (4.241s)
2022-11-23T02:21:02.6419546Z 
2022-11-23T02:21:02.6419817Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6419909Z Ran 1 test in 4.241s
2022-11-23T02:21:02.6419947Z 
2022-11-23T02:21:02.6420021Z OK
2022-11-23T02:21:02.6420039Z 
2022-11-23T02:21:02.6420162Z Generating XML reports...
2022-11-23T02:21:02.6420617Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021615.xml
2022-11-23T02:21:02.6420989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6421164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6421540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6421732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6421751Z 
2022-11-23T02:21:02.6421863Z Running tests...
2022-11-23T02:21:02.6422110Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6422420Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6422682Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6422903Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47352
2022-11-23T02:21:02.6423120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47353
2022-11-23T02:21:02.6423485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6423659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6424036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6424207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6424572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6424744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6425165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6425357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6425603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6425848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6426251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6426643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6427079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6427317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6427562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6427803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6428209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6428599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6428931Z STAGE:2022-11-23 02:16:25 47353:47353 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6429349Z STAGE:2022-11-23 02:16:25 47352:47352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6429680Z STAGE:2022-11-23 02:16:25 47352:47352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6429984Z STAGE:2022-11-23 02:16:25 47353:47353 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6430331Z STAGE:2022-11-23 02:16:25 47352:47352 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6430677Z STAGE:2022-11-23 02:16:25 47353:47353 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6431003Z STAGE:2022-11-23 02:16:25 47353:47353 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6431324Z STAGE:2022-11-23 02:16:25 47352:47352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6431862Z STAGE:2022-11-23 02:16:25 47353:47353 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:16:25 47352:47352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6431887Z 
2022-11-23T02:21:02.6432451Z STAGE:2022-11-23 02:16:25 47353:47353 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:16:25 47352:47352 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6432471Z 
2022-11-23T02:21:02.6432576Z ok (4.375s)
2022-11-23T02:21:02.6432596Z 
2022-11-23T02:21:02.6432860Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6432970Z Ran 1 test in 4.375s
2022-11-23T02:21:02.6432992Z 
2022-11-23T02:21:02.6433083Z OK
2022-11-23T02:21:02.6433103Z 
2022-11-23T02:21:02.6433208Z Generating XML reports...
2022-11-23T02:21:02.6433657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021621.xml
2022-11-23T02:21:02.6434026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6434205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6434585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6434777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6434862Z 
2022-11-23T02:21:02.6434982Z Running tests...
2022-11-23T02:21:02.6435246Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6435540Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6435813Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6436032Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47471
2022-11-23T02:21:02.6436247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47472
2022-11-23T02:21:02.6436618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6436790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6437171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6437359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6437722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6437877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6438245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6438432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6438746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6438990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6439391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6439789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6440019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6440228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6440468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6440707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6441107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6441495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6441830Z STAGE:2022-11-23 02:16:32 47471:47471 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6442153Z STAGE:2022-11-23 02:16:32 47472:47472 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6442486Z STAGE:2022-11-23 02:16:32 47472:47472 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6442812Z STAGE:2022-11-23 02:16:32 47471:47471 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6443157Z STAGE:2022-11-23 02:16:32 47472:47472 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6443486Z STAGE:2022-11-23 02:16:32 47471:47471 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6443815Z STAGE:2022-11-23 02:16:32 47472:47472 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6444137Z STAGE:2022-11-23 02:16:32 47471:47471 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6444518Z STAGE:2022-11-23 02:16:32 47471:47471 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6444853Z STAGE:2022-11-23 02:16:32 47472:47472 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6445412Z STAGE:2022-11-23 02:16:32 47471:47471 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:16:32 47472:47472 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6445432Z 
2022-11-23T02:21:02.6445534Z ok (4.467s)
2022-11-23T02:21:02.6445553Z 
2022-11-23T02:21:02.6445815Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6445930Z Ran 1 test in 4.467s
2022-11-23T02:21:02.6445950Z 
2022-11-23T02:21:02.6446023Z OK
2022-11-23T02:21:02.6446041Z 
2022-11-23T02:21:02.6446165Z Generating XML reports...
2022-11-23T02:21:02.6446616Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021628.xml
2022-11-23T02:21:02.6446988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6447163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6447540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6447731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6447750Z 
2022-11-23T02:21:02.6447858Z Running tests...
2022-11-23T02:21:02.6448116Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6448468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6448731Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6448949Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47590
2022-11-23T02:21:02.6449169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47591
2022-11-23T02:21:02.6449542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6449715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6450085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6450275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6450618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6450793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6451161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6451353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6451601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6451845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6452242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6452633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6452867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6453076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6453317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6453610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6454019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6454408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6454740Z STAGE:2022-11-23 02:16:39 47590:47590 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6455060Z STAGE:2022-11-23 02:16:39 47591:47591 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6455396Z STAGE:2022-11-23 02:16:39 47591:47591 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6455719Z STAGE:2022-11-23 02:16:39 47590:47590 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6456050Z STAGE:2022-11-23 02:16:39 47591:47591 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6456397Z STAGE:2022-11-23 02:16:39 47590:47590 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6456723Z STAGE:2022-11-23 02:16:39 47591:47591 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6457042Z STAGE:2022-11-23 02:16:39 47590:47590 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6457417Z STAGE:2022-11-23 02:16:39 47590:47590 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6457753Z STAGE:2022-11-23 02:16:39 47591:47591 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6458156Z STAGE:2022-11-23 02:16:39 47590:47590 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6458495Z STAGE:2022-11-23 02:16:39 47591:47591 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6458596Z ok (4.223s)
2022-11-23T02:21:02.6458615Z 
2022-11-23T02:21:02.6458860Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6458975Z Ran 1 test in 4.223s
2022-11-23T02:21:02.6458994Z 
2022-11-23T02:21:02.6459085Z OK
2022-11-23T02:21:02.6459106Z 
2022-11-23T02:21:02.6459228Z Generating XML reports...
2022-11-23T02:21:02.6459677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021635.xml
2022-11-23T02:21:02.6460044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6460218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6460602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6460775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6460813Z 
2022-11-23T02:21:02.6460902Z Running tests...
2022-11-23T02:21:02.6461168Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6461479Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6461738Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6461956Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47709
2022-11-23T02:21:02.6462171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47710
2022-11-23T02:21:02.6462539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6462719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6463081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6463321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6463694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6463867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6464237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6464424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6464670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6464919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6465299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6465697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6465928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6466157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6466313Z skip: Skipped due to small world size. (4.254s)
2022-11-23T02:21:02.6466333Z 
2022-11-23T02:21:02.6466594Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6466706Z Ran 1 test in 4.254s
2022-11-23T02:21:02.6466725Z 
2022-11-23T02:21:02.6467070Z OK (skipped=1)
2022-11-23T02:21:02.6467169Z 
2022-11-23T02:21:02.6467304Z Generating XML reports...
2022-11-23T02:21:02.6467751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021642.xml
2022-11-23T02:21:02.6468120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6468297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6468670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6468859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6468878Z 
2022-11-23T02:21:02.6468988Z Running tests...
2022-11-23T02:21:02.6469253Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6469565Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6469824Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6470024Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47818
2022-11-23T02:21:02.6470243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47819
2022-11-23T02:21:02.6470616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6470792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6471171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6471359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6471719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6471891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6472253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6472442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6472751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6473005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6473406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6473797Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6474027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6474256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6474417Z skip: Skipped due to small world size. (4.254s)
2022-11-23T02:21:02.6474437Z 
2022-11-23T02:21:02.6474680Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6474792Z Ran 1 test in 4.254s
2022-11-23T02:21:02.6474811Z 
2022-11-23T02:21:02.6474918Z OK (skipped=1)
2022-11-23T02:21:02.6474940Z 
2022-11-23T02:21:02.6475063Z Generating XML reports...
2022-11-23T02:21:02.6475513Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021649.xml
2022-11-23T02:21:02.6475882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6476057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6476432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6476681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6476700Z 
2022-11-23T02:21:02.6476789Z Running tests...
2022-11-23T02:21:02.6477053Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6477366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6477634Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6477852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47927
2022-11-23T02:21:02.6478068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47928
2022-11-23T02:21:02.6478438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6478613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6478979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6479168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6479528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6479706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6480077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6480264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6480513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6480755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6481154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6481536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6481768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6482047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6482212Z skip: Skipped due to small world size. (4.153s)
2022-11-23T02:21:02.6482231Z 
2022-11-23T02:21:02.6482505Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6482617Z Ran 1 test in 4.153s
2022-11-23T02:21:02.6482636Z 
2022-11-23T02:21:02.6482743Z OK (skipped=1)
2022-11-23T02:21:02.6482762Z 
2022-11-23T02:21:02.6482885Z Generating XML reports...
2022-11-23T02:21:02.6483317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021656.xml
2022-11-23T02:21:02.6483690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6483863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6484243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6484434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6484453Z 
2022-11-23T02:21:02.6484562Z Running tests...
2022-11-23T02:21:02.6484824Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6485136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6485391Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6485590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48036
2022-11-23T02:21:02.6485863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48037
2022-11-23T02:21:02.6486238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6486412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6486790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6486977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6487340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6487510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6487858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6488049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6488293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6488535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6488937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6489332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6489562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6489789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6489944Z skip: Skipped due to small world size. (4.259s)
2022-11-23T02:21:02.6489963Z 
2022-11-23T02:21:02.6490214Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6490325Z Ran 1 test in 4.260s
2022-11-23T02:21:02.6490344Z 
2022-11-23T02:21:02.6490451Z OK (skipped=1)
2022-11-23T02:21:02.6490470Z 
2022-11-23T02:21:02.6490593Z Generating XML reports...
2022-11-23T02:21:02.6491094Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021702.xml
2022-11-23T02:21:02.6491479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6491656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6492033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6492225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6492245Z 
2022-11-23T02:21:02.6492335Z Running tests...
2022-11-23T02:21:02.6492599Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6492908Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6493154Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6493376Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48145
2022-11-23T02:21:02.6493595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48146
2022-11-23T02:21:02.6493967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6494139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6494498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6494687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6495107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6495279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6495655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6495842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6496089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6496331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6496729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6497104Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6497338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6497678Z STAGE:2022-11-23 02:17:13 48146:48146 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6497910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6498236Z STAGE:2022-11-23 02:17:13 48145:48145 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6498569Z STAGE:2022-11-23 02:17:13 48146:48146 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6498904Z STAGE:2022-11-23 02:17:13 48145:48145 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6499473Z STAGE:2022-11-23 02:17:13 48146:48146 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:17:13 48145:48145 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6499497Z 
2022-11-23T02:21:02.6499824Z STAGE:2022-11-23 02:17:13 48146:48146 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6500141Z STAGE:2022-11-23 02:17:13 48145:48145 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6500559Z STAGE:2022-11-23 02:17:13 48146:48146 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6500898Z STAGE:2022-11-23 02:17:13 48145:48145 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6501239Z STAGE:2022-11-23 02:17:13 48146:48146 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6501580Z STAGE:2022-11-23 02:17:13 48145:48145 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6501684Z ok (4.336s)
2022-11-23T02:21:02.6501703Z 
2022-11-23T02:21:02.6501964Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6502079Z Ran 1 test in 4.337s
2022-11-23T02:21:02.6502098Z 
2022-11-23T02:21:02.6502191Z OK
2022-11-23T02:21:02.6502210Z 
2022-11-23T02:21:02.6502315Z Generating XML reports...
2022-11-23T02:21:02.6502769Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021709.xml
2022-11-23T02:21:02.6503140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6503317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6503695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6503885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6503904Z 
2022-11-23T02:21:02.6504013Z Running tests...
2022-11-23T02:21:02.6504273Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6504657Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6504887Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6505105Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48258
2022-11-23T02:21:02.6505326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48259
2022-11-23T02:21:02.6505693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6505866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6506245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6506435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6507017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6507192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6507576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6507766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6508014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6508259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6508658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6509052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6509280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6509514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6509834Z STAGE:2022-11-23 02:17:20 48259:48259 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6510231Z STAGE:2022-11-23 02:17:20 48258:48258 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6510583Z STAGE:2022-11-23 02:17:20 48259:48259 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6510913Z STAGE:2022-11-23 02:17:20 48258:48258 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6511256Z STAGE:2022-11-23 02:17:20 48259:48259 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6511598Z STAGE:2022-11-23 02:17:20 48258:48258 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6511922Z STAGE:2022-11-23 02:17:20 48259:48259 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6512247Z STAGE:2022-11-23 02:17:20 48258:48258 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6512793Z STAGE:2022-11-23 02:17:20 48259:48259 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:17:20 48258:48258 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6512815Z 
2022-11-23T02:21:02.6513162Z STAGE:2022-11-23 02:17:20 48259:48259 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6513488Z STAGE:2022-11-23 02:17:20 48258:48258 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6513589Z ok (4.229s)
2022-11-23T02:21:02.6513608Z 
2022-11-23T02:21:02.6513870Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6513981Z Ran 1 test in 4.229s
2022-11-23T02:21:02.6514001Z 
2022-11-23T02:21:02.6514094Z OK
2022-11-23T02:21:02.6514176Z 
2022-11-23T02:21:02.6514308Z Generating XML reports...
2022-11-23T02:21:02.6514766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021716.xml
2022-11-23T02:21:02.6515134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6515294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6515676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6515866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6515885Z 
2022-11-23T02:21:02.6515993Z Running tests...
2022-11-23T02:21:02.6516260Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6516570Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6516854Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-11-23T02:21:02.6516873Z 
2022-11-23T02:21:02.6517133Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6517242Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6517262Z 
2022-11-23T02:21:02.6517350Z OK (skipped=1)
2022-11-23T02:21:02.6517391Z 
2022-11-23T02:21:02.6517495Z Generating XML reports...
2022-11-23T02:21:02.6517941Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021723.xml
2022-11-23T02:21:02.6518313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6518487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6518865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6519059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6519079Z 
2022-11-23T02:21:02.6519187Z Running tests...
2022-11-23T02:21:02.6519448Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6519742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6520056Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6520283Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48404
2022-11-23T02:21:02.6520498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48405
2022-11-23T02:21:02.6520871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6521044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6521419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6521613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6521956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6522133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6522505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6522693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6522942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6523188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6523586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6524043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6524273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6524485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6524821Z STAGE:2022-11-23 02:17:29 48405:48405 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6525140Z STAGE:2022-11-23 02:17:29 48404:48404 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6525681Z STAGE:2022-11-23 02:17:29 48404:48404 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:17:29 48405:48405 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6525702Z 
2022-11-23T02:21:02.6526273Z STAGE:2022-11-23 02:17:29 48404:48404 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:17:29 48405:48405 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6526293Z 
2022-11-23T02:21:02.6526618Z STAGE:2022-11-23 02:17:29 48405:48405 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6526939Z STAGE:2022-11-23 02:17:29 48404:48404 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6527270Z STAGE:2022-11-23 02:17:29 48404:48404 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6527595Z STAGE:2022-11-23 02:17:29 48405:48405 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6527935Z STAGE:2022-11-23 02:17:29 48404:48404 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6528284Z STAGE:2022-11-23 02:17:29 48405:48405 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6528372Z ok (4.134s)
2022-11-23T02:21:02.6528391Z 
2022-11-23T02:21:02.6528653Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6528763Z Ran 1 test in 4.134s
2022-11-23T02:21:02.6528782Z 
2022-11-23T02:21:02.6528869Z OK
2022-11-23T02:21:02.6528888Z 
2022-11-23T02:21:02.6529010Z Generating XML reports...
2022-11-23T02:21:02.6529506Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021725.xml
2022-11-23T02:21:02.6529882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6530052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6530411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6530595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6530618Z 
2022-11-23T02:21:02.6530724Z Running tests...
2022-11-23T02:21:02.6530985Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6531296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6531594Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2022-11-23T02:21:02.6531614Z 
2022-11-23T02:21:02.6531872Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6531984Z Ran 1 test in 0.003s
2022-11-23T02:21:02.6532003Z 
2022-11-23T02:21:02.6532110Z OK (skipped=1)
2022-11-23T02:21:02.6532128Z 
2022-11-23T02:21:02.6532232Z Generating XML reports...
2022-11-23T02:21:02.6532674Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021732.xml
2022-11-23T02:21:02.6533038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6533266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6533647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6533838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6533857Z 
2022-11-23T02:21:02.6533963Z Running tests...
2022-11-23T02:21:02.6534225Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6534516Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6534791Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-11-23T02:21:02.6534810Z 
2022-11-23T02:21:02.6535068Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6535183Z Ran 1 test in 0.003s
2022-11-23T02:21:02.6535203Z 
2022-11-23T02:21:02.6535311Z OK (skipped=1)
2022-11-23T02:21:02.6535329Z 
2022-11-23T02:21:02.6535454Z Generating XML reports...
2022-11-23T02:21:02.6535901Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021734.xml
2022-11-23T02:21:02.6536272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6536446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6536808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6536998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6537017Z 
2022-11-23T02:21:02.6537126Z Running tests...
2022-11-23T02:21:02.6537385Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6537699Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6537944Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6538163Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48583
2022-11-23T02:21:02.6538431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48584
2022-11-23T02:21:02.6538812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6538967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6539344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6539533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6539895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6540069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6540442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6540629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6540879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6541104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6541506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6541898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6542129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6542528Z STAGE:2022-11-23 02:17:40 48584:48584 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6542755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6543086Z STAGE:2022-11-23 02:17:40 48583:48583 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6543628Z STAGE:2022-11-23 02:17:40 48583:48583 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:17:40 48584:48584 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6543648Z 
2022-11-23T02:21:02.6544213Z STAGE:2022-11-23 02:17:40 48583:48583 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2022-11-23 02:17:40 48584:48584 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6544233Z 
2022-11-23T02:21:02.6544559Z STAGE:2022-11-23 02:17:40 48584:48584 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6544885Z STAGE:2022-11-23 02:17:40 48583:48583 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6545199Z STAGE:2022-11-23 02:17:40 48583:48583 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6545533Z STAGE:2022-11-23 02:17:40 48584:48584 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6545878Z STAGE:2022-11-23 02:17:40 48583:48583 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6546219Z STAGE:2022-11-23 02:17:40 48584:48584 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6546319Z ok (4.359s)
2022-11-23T02:21:02.6546338Z 
2022-11-23T02:21:02.6546599Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6546709Z Ran 1 test in 4.360s
2022-11-23T02:21:02.6546728Z 
2022-11-23T02:21:02.6547030Z OK
2022-11-23T02:21:02.6547058Z 
2022-11-23T02:21:02.6547170Z Generating XML reports...
2022-11-23T02:21:02.6547631Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021737.xml
2022-11-23T02:21:02.6547997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6548251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6548647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6548837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6548857Z 
2022-11-23T02:21:02.6548965Z Running tests...
2022-11-23T02:21:02.6549226Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6549538Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6549782Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-11-23T02:21:02.6549820Z 
2022-11-23T02:21:02.6550058Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6550166Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6550185Z 
2022-11-23T02:21:02.6550294Z OK (skipped=1)
2022-11-23T02:21:02.6550316Z 
2022-11-23T02:21:02.6550440Z Generating XML reports...
2022-11-23T02:21:02.6550890Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021743.xml
2022-11-23T02:21:02.6551256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6551431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6551809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6552053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6552072Z 
2022-11-23T02:21:02.6552180Z Running tests...
2022-11-23T02:21:02.6552444Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6552754Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6553022Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-11-23T02:21:02.6553042Z 
2022-11-23T02:21:02.6553298Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6553409Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6553429Z 
2022-11-23T02:21:02.6553535Z OK (skipped=1)
2022-11-23T02:21:02.6553554Z 
2022-11-23T02:21:02.6553676Z Generating XML reports...
2022-11-23T02:21:02.6554106Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021746.xml
2022-11-23T02:21:02.6554478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6554651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6555024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6555216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6555236Z 
2022-11-23T02:21:02.6555342Z Running tests...
2022-11-23T02:21:02.6555602Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6555911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6556151Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6556370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48762
2022-11-23T02:21:02.6556588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48763
2022-11-23T02:21:02.6556956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6557130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6557599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6557797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6558166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6558338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6558690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6558877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6559127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6559371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6559771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6560209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6560440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6560774Z STAGE:2022-11-23 02:17:52 48763:48763 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6560981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6561310Z STAGE:2022-11-23 02:17:52 48762:48762 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6561704Z STAGE:2022-11-23 02:17:52 48763:48763 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6562032Z STAGE:2022-11-23 02:17:52 48762:48762 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6562379Z STAGE:2022-11-23 02:17:52 48763:48763 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6562723Z STAGE:2022-11-23 02:17:52 48762:48762 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6563047Z STAGE:2022-11-23 02:17:52 48763:48763 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6563366Z STAGE:2022-11-23 02:17:52 48762:48762 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6563697Z STAGE:2022-11-23 02:17:52 48762:48762 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6564011Z STAGE:2022-11-23 02:17:52 48763:48763 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6564354Z STAGE:2022-11-23 02:17:52 48762:48762 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6564691Z STAGE:2022-11-23 02:17:52 48763:48763 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6564796Z ok (4.271s)
2022-11-23T02:21:02.6564816Z 
2022-11-23T02:21:02.6565078Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6565188Z Ran 1 test in 4.271s
2022-11-23T02:21:02.6565207Z 
2022-11-23T02:21:02.6565300Z OK
2022-11-23T02:21:02.6565319Z 
2022-11-23T02:21:02.6565441Z Generating XML reports...
2022-11-23T02:21:02.6565892Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021748.xml
2022-11-23T02:21:02.6566246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6566426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6566803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6566994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6567062Z 
2022-11-23T02:21:02.6567179Z Running tests...
2022-11-23T02:21:02.6567440Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6567749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6567992Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6568194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48875
2022-11-23T02:21:02.6568413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48876
2022-11-23T02:21:02.6568786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6568962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6569339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6569531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6569894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6570067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6570437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6570607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6570853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6571166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6571568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6571963Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6572192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6572526Z STAGE:2022-11-23 02:17:59 48876:48876 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6572752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6573084Z STAGE:2022-11-23 02:17:59 48875:48875 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6573405Z STAGE:2022-11-23 02:17:59 48875:48875 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6573729Z STAGE:2022-11-23 02:17:59 48876:48876 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6574072Z STAGE:2022-11-23 02:17:59 48875:48875 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6574417Z STAGE:2022-11-23 02:17:59 48876:48876 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6574943Z STAGE:2022-11-23 02:17:59 48876:48876 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 02:17:59 48875:48875 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6574963Z 
2022-11-23T02:21:02.6575295Z STAGE:2022-11-23 02:17:59 48876:48876 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6575620Z STAGE:2022-11-23 02:17:59 48875:48875 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6575966Z STAGE:2022-11-23 02:17:59 48876:48876 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6576309Z STAGE:2022-11-23 02:17:59 48875:48875 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6576392Z ok (4.351s)
2022-11-23T02:21:02.6576429Z 
2022-11-23T02:21:02.6576720Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6576839Z Ran 1 test in 4.352s
2022-11-23T02:21:02.6576858Z 
2022-11-23T02:21:02.6576951Z OK
2022-11-23T02:21:02.6576970Z 
2022-11-23T02:21:02.6577093Z Generating XML reports...
2022-11-23T02:21:02.6577547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021755.xml
2022-11-23T02:21:02.6577915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6578092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6578474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6578647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6578666Z 
2022-11-23T02:21:02.6578773Z Running tests...
2022-11-23T02:21:02.6579036Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6579348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6579601Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6579820Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48988
2022-11-23T02:21:02.6580035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48989
2022-11-23T02:21:02.6580402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6580616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6580996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6581186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6581550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6581723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6582092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6582279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6582525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6582778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6583161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6583559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6583789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6584017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6584118Z ok (4.366s)
2022-11-23T02:21:02.6584138Z 
2022-11-23T02:21:02.6584401Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6584512Z Ran 1 test in 4.366s
2022-11-23T02:21:02.6584532Z 
2022-11-23T02:21:02.6584624Z OK
2022-11-23T02:21:02.6584644Z 
2022-11-23T02:21:02.6584766Z Generating XML reports...
2022-11-23T02:21:02.6585201Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021802.xml
2022-11-23T02:21:02.6585570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6585744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6586170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6586367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6586387Z 
2022-11-23T02:21:02.6586495Z Running tests...
2022-11-23T02:21:02.6586962Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6587299Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6587540Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6587797Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49097
2022-11-23T02:21:02.6588015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49098
2022-11-23T02:21:02.6588385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6588563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6588942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6589131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6589490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6589663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6590016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6590294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6590539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6590786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6591192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6591589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6591818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6592154Z STAGE:2022-11-23 02:18:13 49098:49098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6592369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6592701Z STAGE:2022-11-23 02:18:13 49097:49097 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6593035Z STAGE:2022-11-23 02:18:13 49097:49097 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6593361Z STAGE:2022-11-23 02:18:13 49098:49098 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6593707Z STAGE:2022-11-23 02:18:13 49097:49097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6594049Z STAGE:2022-11-23 02:18:13 49098:49098 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6594574Z STAGE:2022-11-23 02:18:13 49097:49097 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2022-11-23 02:18:13 49098:49098 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6594599Z 
2022-11-23T02:21:02.6594933Z STAGE:2022-11-23 02:18:13 49098:49098 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6595261Z STAGE:2022-11-23 02:18:13 49097:49097 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6595666Z STAGE:2022-11-23 02:18:13 49098:49098 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6596002Z STAGE:2022-11-23 02:18:13 49097:49097 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6596102Z ok (4.354s)
2022-11-23T02:21:02.6596121Z 
2022-11-23T02:21:02.6596382Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6596493Z Ran 1 test in 4.355s
2022-11-23T02:21:02.6596513Z 
2022-11-23T02:21:02.6596606Z OK
2022-11-23T02:21:02.6596625Z 
2022-11-23T02:21:02.6596749Z Generating XML reports...
2022-11-23T02:21:02.6597200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021809.xml
2022-11-23T02:21:02.6597575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6597748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6598113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6598301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6598320Z 
2022-11-23T02:21:02.6598426Z Running tests...
2022-11-23T02:21:02.6598685Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6598999Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6599251Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T02:21:02.6599270Z 
2022-11-23T02:21:02.6599591Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6599703Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6599723Z 
2022-11-23T02:21:02.6599811Z OK (skipped=1)
2022-11-23T02:21:02.6599849Z 
2022-11-23T02:21:02.6599952Z Generating XML reports...
2022-11-23T02:21:02.6600410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021816.xml
2022-11-23T02:21:02.6600778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6600951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6601326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6601513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6601533Z 
2022-11-23T02:21:02.6601640Z Running tests...
2022-11-23T02:21:02.6601903Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6602193Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6602461Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-11-23T02:21:02.6602480Z 
2022-11-23T02:21:02.6602737Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6602847Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6602866Z 
2022-11-23T02:21:02.6602972Z OK (skipped=1)
2022-11-23T02:21:02.6602991Z 
2022-11-23T02:21:02.6603114Z Generating XML reports...
2022-11-23T02:21:02.6603563Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021818.xml
2022-11-23T02:21:02.6603930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6604104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6604465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6604653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6604671Z 
2022-11-23T02:21:02.6604778Z Running tests...
2022-11-23T02:21:02.6605085Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6605404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6605661Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6605881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49276
2022-11-23T02:21:02.6606098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49277
2022-11-23T02:21:02.6606449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6606627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6607004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6607197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6607560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6607730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6608105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6608293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6608538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6608827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6609230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6609627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6609857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6610084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6610322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6610564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6610961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6611356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6611674Z STAGE:2022-11-23 02:18:24 49277:49277 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6611998Z STAGE:2022-11-23 02:18:24 49276:49276 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6612327Z STAGE:2022-11-23 02:18:24 49276:49276 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6612652Z STAGE:2022-11-23 02:18:24 49277:49277 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6612998Z STAGE:2022-11-23 02:18:24 49276:49276 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6613341Z STAGE:2022-11-23 02:18:24 49277:49277 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6613671Z STAGE:2022-11-23 02:18:24 49277:49277 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6613988Z STAGE:2022-11-23 02:18:24 49276:49276 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6614316Z STAGE:2022-11-23 02:18:24 49277:49277 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6614679Z STAGE:2022-11-23 02:18:24 49276:49276 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6615033Z STAGE:2022-11-23 02:18:24 49277:49277 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6615374Z STAGE:2022-11-23 02:18:24 49276:49276 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6615475Z ok (4.242s)
2022-11-23T02:21:02.6615495Z 
2022-11-23T02:21:02.6615755Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6615868Z Ran 1 test in 4.242s
2022-11-23T02:21:02.6615891Z 
2022-11-23T02:21:02.6615984Z OK
2022-11-23T02:21:02.6616003Z 
2022-11-23T02:21:02.6616125Z Generating XML reports...
2022-11-23T02:21:02.6616562Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021820.xml
2022-11-23T02:21:02.6616937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6617112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6617488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6617678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6617697Z 
2022-11-23T02:21:02.6617808Z Running tests...
2022-11-23T02:21:02.6618066Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6618377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6618689Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6618891Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49395
2022-11-23T02:21:02.6619107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49396
2022-11-23T02:21:02.6619484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6619657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6620032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6620220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6620579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6620754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6621108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6621299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6621547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6621791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6622188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6622581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6622808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6623039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6623196Z skip: Skipped due to small world size. (4.254s)
2022-11-23T02:21:02.6623218Z 
2022-11-23T02:21:02.6623463Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6623577Z Ran 1 test in 4.254s
2022-11-23T02:21:02.6623599Z 
2022-11-23T02:21:02.6623752Z OK (skipped=1)
2022-11-23T02:21:02.6623774Z 
2022-11-23T02:21:02.6623900Z Generating XML reports...
2022-11-23T02:21:02.6624353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021827.xml
2022-11-23T02:21:02.6624722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6624894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6625269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6625464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6625483Z 
2022-11-23T02:21:02.6625572Z Running tests...
2022-11-23T02:21:02.6625833Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6626145Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6626405Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6626624Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49504
2022-11-23T02:21:02.6627260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49505
2022-11-23T02:21:02.6627657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6627831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6628296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6628486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6628850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6629021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6629396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6629584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6629829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6630074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6630475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6630850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6631087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6631314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6631415Z ok (4.124s)
2022-11-23T02:21:02.6631435Z 
2022-11-23T02:21:02.6631700Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6631810Z Ran 1 test in 4.124s
2022-11-23T02:21:02.6631830Z 
2022-11-23T02:21:02.6631926Z OK
2022-11-23T02:21:02.6631945Z 
2022-11-23T02:21:02.6632067Z Generating XML reports...
2022-11-23T02:21:02.6632498Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021834.xml
2022-11-23T02:21:02.6632870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6633045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6633491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6633694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6633713Z 
2022-11-23T02:21:02.6633820Z Running tests...
2022-11-23T02:21:02.6634084Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6634394Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6634639Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6634839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49613
2022-11-23T02:21:02.6635057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49614
2022-11-23T02:21:02.6635430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6635603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6635978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6636166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6636528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6636701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6637071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6637312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6637557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6637805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6638213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6638608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6638834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6639063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6639165Z ok (4.334s)
2022-11-23T02:21:02.6639185Z 
2022-11-23T02:21:02.6639432Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6639549Z Ran 1 test in 4.334s
2022-11-23T02:21:02.6639568Z 
2022-11-23T02:21:02.6639659Z OK
2022-11-23T02:21:02.6639678Z 
2022-11-23T02:21:02.6639801Z Generating XML reports...
2022-11-23T02:21:02.6640253Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021841.xml
2022-11-23T02:21:02.6640626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6640802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6641183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6641373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6641392Z 
2022-11-23T02:21:02.6641481Z Running tests...
2022-11-23T02:21:02.6641742Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6642056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6642321Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6642540Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49722
2022-11-23T02:21:02.6642803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49723
2022-11-23T02:21:02.6643183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6643355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6643714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6643905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6644272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6644443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6644811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6645001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6645248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6645491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6645891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6646270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6646603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6646832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6646934Z ok (4.166s)
2022-11-23T02:21:02.6646953Z 
2022-11-23T02:21:02.6647219Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6647335Z Ran 1 test in 4.166s
2022-11-23T02:21:02.6647354Z 
2022-11-23T02:21:02.6647450Z OK
2022-11-23T02:21:02.6647469Z 
2022-11-23T02:21:02.6647591Z Generating XML reports...
2022-11-23T02:21:02.6648021Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021848.xml
2022-11-23T02:21:02.6648390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6648563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6648941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6649129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6649148Z 
2022-11-23T02:21:02.6649258Z Running tests...
2022-11-23T02:21:02.6649520Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6649832Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6650120Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6650321Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49831
2022-11-23T02:21:02.6650535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49832
2022-11-23T02:21:02.6650901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6651078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6651455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6651648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6652061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6652239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6652620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6652790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6653036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6653277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6653682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6654075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6654305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6654534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6654870Z STAGE:2022-11-23 02:18:58 49831:49831 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6655181Z STAGE:2022-11-23 02:18:58 49832:49832 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6655511Z STAGE:2022-11-23 02:18:58 49831:49831 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6655916Z STAGE:2022-11-23 02:18:58 49831:49831 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6656244Z STAGE:2022-11-23 02:18:58 49832:49832 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6656588Z STAGE:2022-11-23 02:18:58 49832:49832 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6656693Z ok (4.158s)
2022-11-23T02:21:02.6656712Z 
2022-11-23T02:21:02.6656972Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6657083Z Ran 1 test in 4.159s
2022-11-23T02:21:02.6657103Z 
2022-11-23T02:21:02.6657195Z OK
2022-11-23T02:21:02.6657250Z 
2022-11-23T02:21:02.6657359Z Generating XML reports...
2022-11-23T02:21:02.6657815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021854.xml
2022-11-23T02:21:02.6658183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6658364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6658742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6658932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6658954Z 
2022-11-23T02:21:02.6659063Z Running tests...
2022-11-23T02:21:02.6659326Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6659635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6659899Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6660115Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49944
2022-11-23T02:21:02.6660330Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49945
2022-11-23T02:21:02.6660704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6660877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6661305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6661500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6661866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6662019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6662391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6662576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6662825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6663068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6663465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6663861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6664089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6664311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6664629Z STAGE:2022-11-23 02:19:05 49944:49944 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6664953Z STAGE:2022-11-23 02:19:05 49945:49945 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6665349Z STAGE:2022-11-23 02:19:05 49944:49944 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6665902Z STAGE:2022-11-23 02:19:05 49945:49945 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2022-11-23 02:19:05 49944:49944 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6665922Z 
2022-11-23T02:21:02.6666268Z STAGE:2022-11-23 02:19:05 49945:49945 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6666370Z ok (4.264s)
2022-11-23T02:21:02.6666389Z 
2022-11-23T02:21:02.6666649Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6666968Z Ran 1 test in 4.264s
2022-11-23T02:21:02.6666992Z 
2022-11-23T02:21:02.6667097Z OK
2022-11-23T02:21:02.6667116Z 
2022-11-23T02:21:02.6667223Z Generating XML reports...
2022-11-23T02:21:02.6667680Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021901.xml
2022-11-23T02:21:02.6668054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6668229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6668609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6668800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6668819Z 
2022-11-23T02:21:02.6668926Z Running tests...
2022-11-23T02:21:02.6669188Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6669501Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6669758Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6669977Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50057
2022-11-23T02:21:02.6670195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50058
2022-11-23T02:21:02.6670567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6670740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6671185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6671369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6671747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6671919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6672295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6672489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6672738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6672980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6673383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6673778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6674006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6674342Z STAGE:2022-11-23 02:19:12 50058:50058 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6674554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6674964Z STAGE:2022-11-23 02:19:12 50057:50057 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6675296Z STAGE:2022-11-23 02:19:12 50058:50058 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6675646Z STAGE:2022-11-23 02:19:12 50058:50058 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6675980Z STAGE:2022-11-23 02:19:12 50057:50057 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6676324Z STAGE:2022-11-23 02:19:12 50057:50057 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6676426Z ok (4.348s)
2022-11-23T02:21:02.6676446Z 
2022-11-23T02:21:02.6676707Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6676818Z Ran 1 test in 4.348s
2022-11-23T02:21:02.6676837Z 
2022-11-23T02:21:02.6676910Z OK
2022-11-23T02:21:02.6676929Z 
2022-11-23T02:21:02.6677060Z Generating XML reports...
2022-11-23T02:21:02.6677510Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021908.xml
2022-11-23T02:21:02.6677877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6678055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6678431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6678622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6678641Z 
2022-11-23T02:21:02.6678749Z Running tests...
2022-11-23T02:21:02.6678992Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6679303Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6679540Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T02:21:02.6679563Z 
2022-11-23T02:21:02.6679823Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6679933Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6679952Z 
2022-11-23T02:21:02.6680059Z OK (skipped=1)
2022-11-23T02:21:02.6680078Z 
2022-11-23T02:21:02.6680252Z Generating XML reports...
2022-11-23T02:21:02.6680713Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021915.xml
2022-11-23T02:21:02.6681082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6681236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6681609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6681800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6681824Z 
2022-11-23T02:21:02.6681931Z Running tests...
2022-11-23T02:21:02.6682191Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6682501Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6682770Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T02:21:02.6682790Z 
2022-11-23T02:21:02.6683046Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6683157Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6683175Z 
2022-11-23T02:21:02.6683263Z OK (skipped=1)
2022-11-23T02:21:02.6683282Z 
2022-11-23T02:21:02.6683405Z Generating XML reports...
2022-11-23T02:21:02.6683850Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021917.xml
2022-11-23T02:21:02.6684214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6684444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6684823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6685017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6685036Z 
2022-11-23T02:21:02.6685144Z Running tests...
2022-11-23T02:21:02.6685383Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6685694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6685954Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-11-23T02:21:02.6685973Z 
2022-11-23T02:21:02.6686227Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6686341Z Ran 1 test in 0.002s
2022-11-23T02:21:02.6686360Z 
2022-11-23T02:21:02.6686467Z OK (skipped=1)
2022-11-23T02:21:02.6686485Z 
2022-11-23T02:21:02.6686608Z Generating XML reports...
2022-11-23T02:21:02.6687055Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021919.xml
2022-11-23T02:21:02.6687425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6687582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6687961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6688151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6688170Z 
2022-11-23T02:21:02.6688279Z Running tests...
2022-11-23T02:21:02.6688538Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6688853Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6689120Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6689340Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50269
2022-11-23T02:21:02.6689610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50270
2022-11-23T02:21:02.6689970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6690147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6690519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6690707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6691065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6691239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6691609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6691797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6692025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6692269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6692668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6693055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6693288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6693694Z STAGE:2022-11-23 02:19:26 50270:50270 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6693919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6694243Z STAGE:2022-11-23 02:19:26 50269:50269 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6694576Z STAGE:2022-11-23 02:19:26 50270:50270 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6694918Z STAGE:2022-11-23 02:19:26 50270:50270 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6695247Z STAGE:2022-11-23 02:19:26 50269:50269 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6695588Z STAGE:2022-11-23 02:19:26 50269:50269 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6695671Z ok (4.342s)
2022-11-23T02:21:02.6695695Z 
2022-11-23T02:21:02.6695953Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6696062Z Ran 1 test in 4.342s
2022-11-23T02:21:02.6696081Z 
2022-11-23T02:21:02.6696171Z OK
2022-11-23T02:21:02.6696190Z 
2022-11-23T02:21:02.6696312Z Generating XML reports...
2022-11-23T02:21:02.6696763Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021922.xml
2022-11-23T02:21:02.6697133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6697304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6697660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6697847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6697866Z 
2022-11-23T02:21:02.6697974Z Running tests...
2022-11-23T02:21:02.6698233Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6698542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6698795Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6699059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50382
2022-11-23T02:21:02.6699286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50383
2022-11-23T02:21:02.6699657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6699812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6700186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6700373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6700824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6700994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6701372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6701560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6701803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6702026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6702424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6702817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6703116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6703342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6703441Z ok (4.153s)
2022-11-23T02:21:02.6703461Z 
2022-11-23T02:21:02.6703730Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6703840Z Ran 1 test in 4.153s
2022-11-23T02:21:02.6703860Z 
2022-11-23T02:21:02.6703949Z OK
2022-11-23T02:21:02.6703968Z 
2022-11-23T02:21:02.6704073Z Generating XML reports...
2022-11-23T02:21:02.6704521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021929.xml
2022-11-23T02:21:02.6704883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6705058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6705434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6705623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6705642Z 
2022-11-23T02:21:02.6705746Z Running tests...
2022-11-23T02:21:02.6706008Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6706319Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6706588Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6707016Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50491
2022-11-23T02:21:02.6707244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50492
2022-11-23T02:21:02.6707617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6707796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6708169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6708436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6708812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6708967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6709341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6709525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6709767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6710012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6710408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6710804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6711033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6711367Z STAGE:2022-11-23 02:19:39 50492:50492 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6711578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6711906Z STAGE:2022-11-23 02:19:39 50491:50491 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6712236Z STAGE:2022-11-23 02:19:39 50491:50491 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6712657Z STAGE:2022-11-23 02:19:39 50491:50491 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6712990Z STAGE:2022-11-23 02:19:39 50492:50492 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6713335Z STAGE:2022-11-23 02:19:39 50492:50492 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6713437Z ok (4.318s)
2022-11-23T02:21:02.6713457Z 
2022-11-23T02:21:02.6713721Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6713829Z Ran 1 test in 4.319s
2022-11-23T02:21:02.6713849Z 
2022-11-23T02:21:02.6713923Z OK
2022-11-23T02:21:02.6713942Z 
2022-11-23T02:21:02.6714064Z Generating XML reports...
2022-11-23T02:21:02.6714509Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021935.xml
2022-11-23T02:21:02.6714881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6715052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6715426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6715617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6715636Z 
2022-11-23T02:21:02.6715742Z Running tests...
2022-11-23T02:21:02.6715982Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6716293Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6716569Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6716785Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50604
2022-11-23T02:21:02.6717002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50605
2022-11-23T02:21:02.6717368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6717540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6717964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6718162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6718511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6718682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6719051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6719236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6719483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6719722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6720123Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6720517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6720744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6720955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6721288Z STAGE:2022-11-23 02:19:46 50604:50604 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6721610Z STAGE:2022-11-23 02:19:46 50605:50605 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2022-11-23T02:21:02.6722008Z STAGE:2022-11-23 02:19:46 50604:50604 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6722355Z STAGE:2022-11-23 02:19:46 50604:50604 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6722690Z STAGE:2022-11-23 02:19:46 50605:50605 ActivityProfilerController.cpp:306] Completed Stage: Collection
2022-11-23T02:21:02.6723036Z STAGE:2022-11-23 02:19:46 50605:50605 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2022-11-23T02:21:02.6723136Z ok (4.175s)
2022-11-23T02:21:02.6723156Z 
2022-11-23T02:21:02.6723401Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6723510Z Ran 1 test in 4.175s
2022-11-23T02:21:02.6723529Z 
2022-11-23T02:21:02.6723618Z OK
2022-11-23T02:21:02.6723636Z 
2022-11-23T02:21:02.6723759Z Generating XML reports...
2022-11-23T02:21:02.6724209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021942.xml
2022-11-23T02:21:02.6724571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6724744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6725123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6725311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6725331Z 
2022-11-23T02:21:02.6725422Z Running tests...
2022-11-23T02:21:02.6725682Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6725989Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6726250Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6726468Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50717
2022-11-23T02:21:02.6726682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50718
2022-11-23T02:21:02.6727049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6727272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6727645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6727835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6728191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6728361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6728729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6728920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6729162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6729406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6729805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6730184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6730413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6730637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6730791Z ok (4.348s)
2022-11-23T02:21:02.6730811Z 
2022-11-23T02:21:02.6731077Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6731188Z Ran 1 test in 4.349s
2022-11-23T02:21:02.6731207Z 
2022-11-23T02:21:02.6731300Z OK
2022-11-23T02:21:02.6731319Z 
2022-11-23T02:21:02.6731442Z Generating XML reports...
2022-11-23T02:21:02.6731876Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021949.xml
2022-11-23T02:21:02.6732244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6732417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6732790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6733051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6733071Z 
2022-11-23T02:21:02.6733181Z Running tests...
2022-11-23T02:21:02.6733440Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6733746Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6734013Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6734214Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50916
2022-11-23T02:21:02.6734428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50917
2022-11-23T02:21:02.6734794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6734966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6735336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6735526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6735882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6736052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6736458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6736652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6736896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6737140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6737542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6737934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6738167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6738394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6738492Z ok (5.782s)
2022-11-23T02:21:02.6738515Z 
2022-11-23T02:21:02.6738760Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6738869Z Ran 1 test in 5.783s
2022-11-23T02:21:02.6738888Z 
2022-11-23T02:21:02.6738977Z OK
2022-11-23T02:21:02.6738997Z 
2022-11-23T02:21:02.6739117Z Generating XML reports...
2022-11-23T02:21:02.6739563Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021956.xml
2022-11-23T02:21:02.6739928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6740156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6740534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6740723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6740743Z 
2022-11-23T02:21:02.6740836Z Running tests...
2022-11-23T02:21:02.6741094Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6741402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6741665Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6741883Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51117
2022-11-23T02:21:02.6742096Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51118
2022-11-23T02:21:02.6742460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6742635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6742995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6743188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6743550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6743721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6744089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6744273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6744517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6744761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6745160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6745587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6745822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6746049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6746150Z ok (6.254s)
2022-11-23T02:21:02.6746170Z 
2022-11-23T02:21:02.6746435Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6746542Z Ran 1 test in 6.254s
2022-11-23T02:21:02.6746561Z 
2022-11-23T02:21:02.6746653Z OK
2022-11-23T02:21:02.6746676Z 
2022-11-23T02:21:02.6747001Z Generating XML reports...
2022-11-23T02:21:02.6747452Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022004.xml
2022-11-23T02:21:02.6747822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6748001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6748379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6748567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6748587Z 
2022-11-23T02:21:02.6748692Z Running tests...
2022-11-23T02:21:02.6748951Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6749261Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6749613Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6749815Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51232
2022-11-23T02:21:02.6750033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51233
2022-11-23T02:21:02.6750405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6750575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6750951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6751139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6751495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6751665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6752022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6752206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6752454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6752697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6753094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6753485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6753712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6753969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqychwrwu
2022-11-23T02:21:02.6754245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqychwrwu/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6754454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6754770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3avhv799
2022-11-23T02:21:02.6755048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3avhv799/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6755147Z ok (4.368s)
2022-11-23T02:21:02.6755166Z 
2022-11-23T02:21:02.6755434Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6755545Z Ran 1 test in 4.369s
2022-11-23T02:21:02.6755564Z 
2022-11-23T02:21:02.6755656Z OK
2022-11-23T02:21:02.6755675Z 
2022-11-23T02:21:02.6755797Z Generating XML reports...
2022-11-23T02:21:02.6756240Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022013.xml
2022-11-23T02:21:02.6756595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6756765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6757144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6757374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6757394Z 
2022-11-23T02:21:02.6757502Z Running tests...
2022-11-23T02:21:02.6757766Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6758076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6758325Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6758526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51345
2022-11-23T02:21:02.6758802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51346
2022-11-23T02:21:02.6759175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6759353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6759729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6759998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6760355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6760528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6760895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6761067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6761312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6761553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6761953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6762342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6762570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6762790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6763042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzg943x7
2022-11-23T02:21:02.6763313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzg943x7/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6763549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp96uqyaw
2022-11-23T02:21:02.6763865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp96uqyaw/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6763973Z ok (5.759s)
2022-11-23T02:21:02.6763992Z 
2022-11-23T02:21:02.6764261Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6764371Z Ran 1 test in 5.760s
2022-11-23T02:21:02.6764391Z 
2022-11-23T02:21:02.6764484Z OK
2022-11-23T02:21:02.6764502Z 
2022-11-23T02:21:02.6764625Z Generating XML reports...
2022-11-23T02:21:02.6765073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022020.xml
2022-11-23T02:21:02.6765424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6765602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6765977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6766164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6766188Z 
2022-11-23T02:21:02.6766293Z Running tests...
2022-11-23T02:21:02.6766552Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6766861Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6767153Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6767370Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51456
2022-11-23T02:21:02.6767569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51457
2022-11-23T02:21:02.6768010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6768186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6768564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6768751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6769107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6769276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6769646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6769815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6770063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6770304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6770703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6771097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6771321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6771548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6771802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxk7g01c3
2022-11-23T02:21:02.6772068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxk7g01c3/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6772304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kb8hbv8
2022-11-23T02:21:02.6772569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kb8hbv8/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6773394Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:21:02.6774179Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:21:02.6774290Z ok (6.215s)
2022-11-23T02:21:02.6774310Z 
2022-11-23T02:21:02.6774579Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6774689Z Ran 1 test in 6.216s
2022-11-23T02:21:02.6774708Z 
2022-11-23T02:21:02.6774801Z OK
2022-11-23T02:21:02.6774819Z 
2022-11-23T02:21:02.6774942Z Generating XML reports...
2022-11-23T02:21:02.6775389Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022028.xml
2022-11-23T02:21:02.6775755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6775983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6776344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6776533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6776554Z 
2022-11-23T02:21:02.6776663Z Running tests...
2022-11-23T02:21:02.6776919Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6777228Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6777511Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6777728Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51571
2022-11-23T02:21:02.6777940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51572
2022-11-23T02:21:02.6778315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6778472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6778845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6779036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6779396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6779565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6779929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6780114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6780356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6780585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6780984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6781463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6781699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6781930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6782168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6782410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6782810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6783198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6783424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:21:02.6783660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:21:02.6784048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.6784430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.6784687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkstp7hyo
2022-11-23T02:21:02.6784959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkstp7hyo/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6785268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8e1twkx1
2022-11-23T02:21:02.6785535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8e1twkx1/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6785639Z ok (10.834s)
2022-11-23T02:21:02.6785661Z 
2022-11-23T02:21:02.6785914Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6786024Z Ran 1 test in 10.834s
2022-11-23T02:21:02.6786044Z 
2022-11-23T02:21:02.6786134Z OK
2022-11-23T02:21:02.6786154Z 
2022-11-23T02:21:02.6786277Z Generating XML reports...
2022-11-23T02:21:02.6786724Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022037.xml
2022-11-23T02:21:02.6787329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6787512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6787894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6788085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6788106Z 
2022-11-23T02:21:02.6788199Z Running tests...
2022-11-23T02:21:02.6788460Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6788770Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-11-23T02:21:02.6789053Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:21:02.6789269Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51694
2022-11-23T02:21:02.6789482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51695
2022-11-23T02:21:02.6789855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6790026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6790387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6790698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6791080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:21:02.6791248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:21:02.6791614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:21:02.6791801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:21:02.6792047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:21:02.6792292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:21:02.6792689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6793073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:21:02.6793299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:21:02.6793525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:21:02.6793761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-11-23T02:21:02.6793999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-11-23T02:21:02.6794394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6794865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-11-23T02:21:02.6795105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-11-23T02:21:02.6795344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-11-23T02:21:02.6795713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.6796104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-11-23T02:21:02.6796358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqcba2log
2022-11-23T02:21:02.6796626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqcba2log/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6796879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3dp3_dh
2022-11-23T02:21:02.6797144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3dp3_dh/_remote_module_non_scriptable.py
2022-11-23T02:21:02.6797243Z ok (10.952s)
2022-11-23T02:21:02.6797263Z 
2022-11-23T02:21:02.6797531Z ----------------------------------------------------------------------
2022-11-23T02:21:02.6797641Z Ran 1 test in 10.952s
2022-11-23T02:21:02.6797661Z 
2022-11-23T02:21:02.6797736Z OK
2022-11-23T02:21:02.6797754Z 
2022-11-23T02:21:02.6797873Z Generating XML reports...
2022-11-23T02:21:02.6798322Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022050.xml
2022-11-23T02:21:02.6798341Z 
2022-11-23T02:21:02.6798760Z ##[endgroup]
2022-11-23T02:21:02.6799225Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_pvjooool)
2022-11-23T02:21:02.6799249Z 
2022-11-23T02:21:02.6799373Z Shard 1: ucc should be run in 3
2022-11-23T02:21:02.6799637Z Running distributed/rpc/test_faulty_agent ... [2022-11-23 02:21:02.404800]
2022-11-23T02:21:02.6800183Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:21:02.405152]
2022-11-23T02:21:04.9373370Z 
2022-11-23T02:21:04.9373841Z Expand the folded group to see the log file of distributed/rpc/test_faulty_agent
2022-11-23T02:21:04.9374813Z ##[group]PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent_6fztm0ks)
2022-11-23T02:21:04.9375403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_1slo0f
2022-11-23T02:21:04.9375929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_1slo0f/_remote_module_non_scriptable.py
2022-11-23T02:21:04.9376249Z 
2022-11-23T02:21:04.9376546Z ##[endgroup]
2022-11-23T02:21:04.9377281Z FINISHED PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent_6fztm0ks)
2022-11-23T02:21:04.9377630Z 
2022-11-23T02:21:04.9378584Z Running distributed/pipeline/sync/test_stream ... [2022-11-23 02:21:04.937479]
2022-11-23T02:21:04.9382139Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2022-11-23 02:21:04.937772]
2022-11-23T02:21:10.3611403Z 
2022-11-23T02:21:10.3612145Z Expand the folded group to see the log file of distributed/pipeline/sync/test_stream
2022-11-23T02:21:10.3613963Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_ff81ma10)
2022-11-23T02:21:10.3614545Z ============================= test session starts ==============================
2022-11-23T02:21:10.3615408Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:21:10.3615761Z cachedir: .pytest_cache
2022-11-23T02:21:10.3616310Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:21:10.3616749Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:21:10.3617071Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:21:10.3617620Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:21:10.3618021Z collecting ... collected 19 items
2022-11-23T02:21:10.3620501Z Running 19 items in this shard: test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view
2022-11-23T02:21:10.3623031Z 
2022-11-23T02:21:10.3623282Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [  5%]
2022-11-23T02:21:10.3623746Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda PASSED [ 10%]
2022-11-23T02:21:10.3624237Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%]
2022-11-23T02:21:10.3624741Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda PASSED [ 21%]
2022-11-23T02:21:10.3625223Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%]
2022-11-23T02:21:10.3625729Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda PASSED [ 31%]
2022-11-23T02:21:10.3626212Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%]
2022-11-23T02:21:10.3626683Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda PASSED [ 42%]
2022-11-23T02:21:10.3627430Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%]
2022-11-23T02:21:10.3627910Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda PASSED [ 52%]
2022-11-23T02:21:10.3628381Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%]
2022-11-23T02:21:10.3628831Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda PASSED [ 63%]
2022-11-23T02:21:10.3629312Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%]
2022-11-23T02:21:10.3629802Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda PASSED [ 73%]
2022-11-23T02:21:10.3630395Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu PASSED [ 78%]
2022-11-23T02:21:10.3630868Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda PASSED [ 84%]
2022-11-23T02:21:10.3631368Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%]
2022-11-23T02:21:10.3631859Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda PASSED [ 94%]
2022-11-23T02:21:10.3632364Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view PASSED [100%]
2022-11-23T02:21:10.3632634Z 
2022-11-23T02:21:10.3632792Z ============================== 19 passed in 2.50s ==============================
2022-11-23T02:21:10.3632985Z 
2022-11-23T02:21:10.3633312Z ##[endgroup]
2022-11-23T02:21:10.3633964Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_ff81ma10)
2022-11-23T02:21:10.3634347Z 
2022-11-23T02:21:10.3634609Z Running distributed/pipeline/sync/test_phony ... [2022-11-23 02:21:10.361125]
2022-11-23T02:21:10.3635218Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_phony.py', '-v'] ... [2022-11-23 02:21:10.361385]
2022-11-23T02:21:12.8165206Z 
2022-11-23T02:21:12.8165956Z Expand the folded group to see the log file of distributed/pipeline/sync/test_phony
2022-11-23T02:21:12.8166910Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_0vhmpt27)
2022-11-23T02:21:12.8167640Z ============================= test session starts ==============================
2022-11-23T02:21:12.8168270Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:21:12.8168861Z cachedir: .pytest_cache
2022-11-23T02:21:12.8169593Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:21:12.8170045Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:21:12.8170371Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:21:12.8171383Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:21:12.8171791Z collecting ... collected 4 items
2022-11-23T02:21:12.8172474Z Running 4 items in this shard: test/distributed/pipeline/sync/test_phony.py::test_phony_size, test/distributed/pipeline/sync/test_phony.py::test_phony_requires_grad, test/distributed/pipeline/sync/test_phony.py::test_cached_phony, test/distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function
2022-11-23T02:21:12.8173006Z 
2022-11-23T02:21:12.8173215Z distributed/pipeline/sync/test_phony.py::test_phony_size PASSED          [ 25%]
2022-11-23T02:21:12.8173645Z distributed/pipeline/sync/test_phony.py::test_phony_requires_grad PASSED [ 50%]
2022-11-23T02:21:12.8174084Z distributed/pipeline/sync/test_phony.py::test_cached_phony PASSED        [ 75%]
2022-11-23T02:21:12.8174509Z distributed/pipeline/sync/test_phony.py::test_phony_in_autograd_function PASSED [100%]
2022-11-23T02:21:12.8174767Z 
2022-11-23T02:21:12.8174925Z ============================== 4 passed in 0.06s ===============================
2022-11-23T02:21:12.8175115Z 
2022-11-23T02:21:12.8175426Z ##[endgroup]
2022-11-23T02:21:12.8176044Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_phony (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_phony_0vhmpt27)
2022-11-23T02:21:12.8176424Z 
2022-11-23T02:21:12.8176720Z Running distributed/pipeline/sync/test_dependency ... [2022-11-23 02:21:12.816547]
2022-11-23T02:21:12.8177348Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2022-11-23 02:21:12.816804]
2022-11-23T02:21:16.6043947Z 
2022-11-23T02:21:16.6044740Z Expand the folded group to see the log file of distributed/pipeline/sync/test_dependency
2022-11-23T02:21:16.6045855Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency_55_0nv62)
2022-11-23T02:21:16.6046401Z ============================= test session starts ==============================
2022-11-23T02:21:16.6047011Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:21:16.6047386Z cachedir: .pytest_cache
2022-11-23T02:21:16.6047956Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:21:16.6048376Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:21:16.6048694Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:21:16.6049265Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:21:16.6049665Z collecting ... collected 6 items
2022-11-23T02:21:16.6050573Z Running 6 items in this shard: test/distributed/pipeline/sync/test_dependency.py::test_fork_join, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_leak, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad
2022-11-23T02:21:16.6051344Z 
2022-11-23T02:21:16.6051561Z distributed/pipeline/sync/test_dependency.py::test_fork_join PASSED      [ 16%]
2022-11-23T02:21:16.6052021Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%]
2022-11-23T02:21:16.6052479Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%]
2022-11-23T02:21:16.6052910Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED      [ 66%]
2022-11-23T02:21:16.6053382Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%]
2022-11-23T02:21:16.6053871Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%]
2022-11-23T02:21:16.6054138Z 
2022-11-23T02:21:16.6054515Z ============================== 6 passed in 1.36s ===============================
2022-11-23T02:21:16.6054725Z 
2022-11-23T02:21:16.6055016Z ##[endgroup]
2022-11-23T02:21:16.6055683Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency_55_0nv62)
2022-11-23T02:21:16.6056072Z 
2022-11-23T02:21:16.6056358Z Running distributed/pipeline/sync/test_checkpoint ... [2022-11-23 02:21:16.604405]
2022-11-23T02:21:16.6057027Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_checkpoint.py', '-v'] ... [2022-11-23 02:21:16.604663]
2022-11-23T02:21:20.4726197Z 
2022-11-23T02:21:20.4727176Z Expand the folded group to see the log file of distributed/pipeline/sync/test_checkpoint
2022-11-23T02:21:20.4728253Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_0w867i8h)
2022-11-23T02:21:20.4728822Z ============================= test session starts ==============================
2022-11-23T02:21:20.4729470Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:21:20.4729819Z cachedir: .pytest_cache
2022-11-23T02:21:20.4730385Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:21:20.4730817Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:21:20.4731127Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:21:20.4731704Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:21:20.4732370Z collecting ... collected 9 items
2022-11-23T02:21:20.4733700Z Running 9 items in this shard: test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad, test/distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter, test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu], test/distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda], test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing, test/distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint, test/distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output
2022-11-23T02:21:20.4734825Z 
2022-11-23T02:21:20.4735064Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cpu] PASSED [ 11%]
2022-11-23T02:21:20.4735537Z distributed/pipeline/sync/test_checkpoint.py::test_serial_checkpoints[cuda] PASSED [ 22%]
2022-11-23T02:21:20.4736006Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad PASSED [ 33%]
2022-11-23T02:21:20.4736493Z distributed/pipeline/sync/test_checkpoint.py::test_not_requires_grad_with_parameter PASSED [ 44%]
2022-11-23T02:21:20.4736985Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cpu] PASSED [ 55%]
2022-11-23T02:21:20.4737454Z distributed/pipeline/sync/test_checkpoint.py::test_random_in_checkpoint[cuda] PASSED [ 66%]
2022-11-23T02:21:20.4737948Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing PASSED [ 77%]
2022-11-23T02:21:20.4738489Z distributed/pipeline/sync/test_checkpoint.py::test_detect_checkpointing_recomputing_without_checkpoint PASSED [ 88%]
2022-11-23T02:21:20.4739002Z distributed/pipeline/sync/test_checkpoint.py::test_non_grad_output PASSED [100%]
2022-11-23T02:21:20.4739238Z 
2022-11-23T02:21:20.4739398Z ============================== 9 passed in 1.40s ===============================
2022-11-23T02:21:20.4739591Z 
2022-11-23T02:21:20.4739908Z ##[endgroup]
2022-11-23T02:21:20.4740677Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_checkpoint_0w867i8h)
2022-11-23T02:21:20.4741084Z 
2022-11-23T02:21:20.4741398Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2022-11-23 02:21:20.472653]
2022-11-23T02:21:20.4742060Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2022-11-23 02:21:20.472914]
2022-11-23T02:21:22.9238741Z 
2022-11-23T02:21:22.9239276Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_verify_skippables
2022-11-23T02:21:22.9240301Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_rq30dgnw)
2022-11-23T02:21:22.9240890Z ============================= test session starts ==============================
2022-11-23T02:21:22.9241480Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:21:22.9241841Z cachedir: .pytest_cache
2022-11-23T02:21:22.9242397Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:21:22.9242846Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:21:22.9243143Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:21:22.9243707Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:21:22.9244139Z collecting ... collected 9 items
2022-11-23T02:21:22.9245468Z Running 9 items in this shard: test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated
2022-11-23T02:21:22.9246850Z 
2022-11-23T02:21:22.9247083Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%]
2022-11-23T02:21:22.9247544Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%]
2022-11-23T02:21:22.9248034Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%]
2022-11-23T02:21:22.9248507Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%]
2022-11-23T02:21:22.9248976Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%]
2022-11-23T02:21:22.9249469Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%]
2022-11-23T02:21:22.9250003Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%]
2022-11-23T02:21:22.9250499Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%]
2022-11-23T02:21:22.9251010Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%]
2022-11-23T02:21:22.9251282Z 
2022-11-23T02:21:22.9251441Z ============================== 9 passed in 0.05s ===============================
2022-11-23T02:21:22.9251641Z 
2022-11-23T02:21:22.9251944Z ##[endgroup]
2022-11-23T02:21:22.9252660Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_rq30dgnw)
2022-11-23T02:21:22.9253090Z 
2022-11-23T02:21:22.9253461Z Running distributed/pipeline/sync/skip/test_portal ... [2022-11-23 02:21:22.923963]
2022-11-23T02:21:22.9254108Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_portal.py', '-v'] ... [2022-11-23 02:21:22.924215]
2022-11-23T02:21:26.7720101Z 
2022-11-23T02:21:26.7720639Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_portal
2022-11-23T02:21:26.7721657Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_qkjt2uyq)
2022-11-23T02:21:26.7722216Z ============================= test session starts ==============================
2022-11-23T02:21:26.7722845Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:21:26.7723206Z cachedir: .pytest_cache
2022-11-23T02:21:26.7723780Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:21:26.7724220Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:21:26.7724529Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:21:26.7725103Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:21:26.7725502Z collecting ... collected 10 items
2022-11-23T02:21:26.7726879Z Running 10 items in this shard: test/distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange, test/distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad, test/distributed/pipeline/sync/skip/test_portal.py::test_use_grad, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4, test/distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1
2022-11-23T02:21:26.7728326Z 
2022-11-23T02:21:26.7728568Z distributed/pipeline/sync/skip/test_portal.py::test_copy_returns_on_next_device PASSED [ 10%]
2022-11-23T02:21:26.7729050Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange PASSED   [ 20%]
2022-11-23T02:21:26.7729501Z distributed/pipeline/sync/skip/test_portal.py::test_blue_orange_not_requires_grad PASSED [ 30%]
2022-11-23T02:21:26.7729964Z distributed/pipeline/sync/skip/test_portal.py::test_use_grad PASSED      [ 40%]
2022-11-23T02:21:26.7730434Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_0 PASSED [ 50%]
2022-11-23T02:21:26.7730921Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_1 PASSED [ 60%]
2022-11-23T02:21:26.7731392Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_2 PASSED [ 70%]
2022-11-23T02:21:26.7731862Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3 PASSED [ 80%]
2022-11-23T02:21:26.7732337Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_4 PASSED [ 90%]
2022-11-23T02:21:26.7732830Z distributed/pipeline/sync/skip/test_portal.py::TestTensorLife::test_tensor_life_3_plus_1 PASSED [100%]
2022-11-23T02:21:26.7733091Z 
2022-11-23T02:21:26.7733251Z ============================== 10 passed in 1.41s ==============================
2022-11-23T02:21:26.7733443Z 
2022-11-23T02:21:26.7733752Z ##[endgroup]
2022-11-23T02:21:26.7734433Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_portal (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_portal_qkjt2uyq)
2022-11-23T02:21:26.7734830Z 
2022-11-23T02:21:26.7735104Z Running distributed/pipeline/sync/skip/test_gpipe ... [2022-11-23 02:21:26.772058]
2022-11-23T02:21:26.7735824Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2022-11-23 02:21:26.772316]
2022-11-23T02:21:34.5857419Z 
2022-11-23T02:21:34.5858192Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_gpipe
2022-11-23T02:21:34.5859203Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_o04src3v)
2022-11-23T02:21:34.5859766Z ============================= test session starts ==============================
2022-11-23T02:21:34.5860375Z platform linux -- Python 3.10.4, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2022-11-23T02:21:34.5860754Z cachedir: .pytest_cache
2022-11-23T02:21:34.5861307Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-11-23T02:21:34.5861744Z torch: 1.14.0a0+git1cfd385
2022-11-23T02:21:34.5862079Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-11-23T02:21:34.5862634Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.0.2, xdoctest-1.0.2
2022-11-23T02:21:34.5863038Z collecting ... collected 13 items
2022-11-23T02:21:34.5865264Z Running 13 items in this shard: test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip
2022-11-23T02:21:34.5867199Z 
2022-11-23T02:21:34.5867551Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] PASSED  [  7%]
2022-11-23T02:21:34.5868085Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] PASSED [ 15%]
2022-11-23T02:21:34.5868602Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] PASSED [ 23%]
2022-11-23T02:21:34.5869116Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%]
2022-11-23T02:21:34.5869650Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] PASSED [ 38%]
2022-11-23T02:21:34.5870171Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] PASSED [ 46%]
2022-11-23T02:21:34.5870690Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] PASSED [ 53%]
2022-11-23T02:21:34.5871207Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%]
2022-11-23T02:21:34.5871745Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] PASSED [ 69%]
2022-11-23T02:21:34.5872283Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] PASSED [ 76%]
2022-11-23T02:21:34.5872805Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] PASSED [ 84%]
2022-11-23T02:21:34.5873350Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%]
2022-11-23T02:21:34.5873795Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED      [100%]
2022-11-23T02:21:34.5874050Z 
2022-11-23T02:21:34.5874218Z ======================== 10 passed, 3 skipped in 5.02s =========================
2022-11-23T02:21:34.5874419Z 
2022-11-23T02:21:34.5874718Z ##[endgroup]
2022-11-23T02:21:34.5875493Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_o04src3v)
2022-11-23T02:21:34.5875905Z 
2022-11-23T02:21:34.5876175Z Running distributed/fsdp/test_shard_utils ... [2022-11-23 02:21:34.585749]
2022-11-23T02:21:34.5876829Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:21:34.586017]
2022-11-23T02:21:36.7707645Z 
2022-11-23T02:21:36.7708151Z Expand the folded group to see the log file of distributed/fsdp/test_shard_utils
2022-11-23T02:21:36.7709078Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_4lb6mso4)
2022-11-23T02:21:36.7709452Z 
2022-11-23T02:21:36.7709752Z ##[endgroup]
2022-11-23T02:21:36.7710626Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_4lb6mso4)
2022-11-23T02:21:36.7710970Z 
2022-11-23T02:21:36.7711567Z Running distributed/checkpoint/test_traverse ... [2022-11-23 02:21:36.770880]
2022-11-23T02:21:36.7714586Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:21:36.771154]
2022-11-23T02:21:40.7371772Z 
2022-11-23T02:21:40.7372421Z Expand the folded group to see the log file of distributed/checkpoint/test_traverse
2022-11-23T02:21:40.7373887Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_traverse_jyo7n2nf)
2022-11-23T02:21:40.7375161Z 
2022-11-23T02:21:40.7375579Z Running tests...
2022-11-23T02:21:40.7376506Z ----------------------------------------------------------------------
2022-11-23T02:21:40.7377649Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_traverse
2022-11-23T02:21:40.7378475Z   test_get_element (__main__.TestTraverse) ... ok (1.609s)
2022-11-23T02:21:40.7379000Z   test_set_element (__main__.TestTraverse) ... ok (0.002s)
2022-11-23T02:21:40.7379418Z   test_traverse_doesnt_ignore_intermediate_collections (__main__.TestTraverse) ... ok (0.003s)
2022-11-23T02:21:40.7379848Z   test_traverse_nested_dict (__main__.TestTraverse) ... ok (0.001s)
2022-11-23T02:21:40.7380211Z   test_traverse_nested_list (__main__.TestTraverse) ... ok (0.002s)
2022-11-23T02:21:40.7380571Z   test_traverse_shallow (__main__.TestTraverse) ... ok (0.002s)
2022-11-23T02:21:40.7380951Z   test_traverse_with_ordered_dict (__main__.TestTraverse) ... ok (0.001s)
2022-11-23T02:21:40.7381194Z 
2022-11-23T02:21:40.7381683Z ----------------------------------------------------------------------
2022-11-23T02:21:40.7382221Z Ran 7 tests in 1.622s
2022-11-23T02:21:40.7382387Z 
2022-11-23T02:21:40.7382479Z OK
2022-11-23T02:21:40.7382620Z 
2022-11-23T02:21:40.7382743Z Generating XML reports...
2022-11-23T02:21:40.7383352Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_traverse/TEST-TestTraverse-20221123022138.xml
2022-11-23T02:21:40.7383707Z 
2022-11-23T02:21:40.7384066Z ##[endgroup]
2022-11-23T02:21:40.7384690Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_traverse (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_traverse_jyo7n2nf)
2022-11-23T02:21:40.7385064Z 
2022-11-23T02:21:40.7385333Z Running distributed/_shard/test_replicated_tensor ... [2022-11-23 02:21:40.737304]
2022-11-23T02:21:40.7386053Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_replicated_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:21:40.737640]
2022-11-23T02:21:42.8565088Z 
2022-11-23T02:21:42.8565854Z Expand the folded group to see the log file of distributed/_shard/test_replicated_tensor
2022-11-23T02:21:42.8566822Z ##[group]PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_yvapar5p)
2022-11-23T02:21:42.8567460Z 
2022-11-23T02:21:42.8567779Z ##[endgroup]
2022-11-23T02:21:42.8568500Z FINISHED PRINTING LOG FILE of distributed/_shard/test_replicated_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_replicated_tensor_yvapar5p)
2022-11-23T02:21:42.8568871Z 
2022-11-23T02:21:42.8569775Z Running distributed/test_nccl ... [2022-11-23 02:21:42.856703]
2022-11-23T02:21:42.8572920Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_nccl.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:21:42.856999]
2022-11-23T02:21:47.9946645Z 
2022-11-23T02:21:47.9947144Z Expand the folded group to see the log file of distributed/test_nccl
2022-11-23T02:21:47.9948268Z ##[group]PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_i7wadmwq)
2022-11-23T02:21:47.9948590Z 
2022-11-23T02:21:47.9948701Z Running tests...
2022-11-23T02:21:47.9949212Z ----------------------------------------------------------------------
2022-11-23T02:21:47.9949754Z Test results will be stored in test-reports/python-unittest/distributed.test_nccl
2022-11-23T02:21:47.9950190Z   test_all_gather_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (1.145s)
2022-11-23T02:21:47.9950559Z   test_all_gather_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-11-23T02:21:47.9950932Z   test_all_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-11-23T02:21:47.9951318Z   test_all_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-11-23T02:21:47.9951676Z   test_broadcast_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-11-23T02:21:47.9952332Z   test_broadcast_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.007s)
2022-11-23T02:21:47.9952714Z   test_collective_errors_cuda (__main__.TestNCCLCUDA) ... ok (0.002s)
2022-11-23T02:21:47.9953105Z   test_reduce_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-11-23T02:21:47.9953446Z   test_reduce_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.003s)
2022-11-23T02:21:47.9953861Z   test_reduce_scatter_cuda_bfloat16 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-11-23T02:21:47.9954255Z   test_reduce_scatter_cuda_float32 (__main__.TestNCCLCUDA) ... ok (0.004s)
2022-11-23T02:21:47.9954606Z   test_unique_id_cuda (__main__.TestNCCLCUDA) ... ok (0.001s)
2022-11-23T02:21:47.9954817Z 
2022-11-23T02:21:47.9955101Z ----------------------------------------------------------------------
2022-11-23T02:21:47.9955434Z Ran 12 tests in 1.183s
2022-11-23T02:21:47.9955669Z 
2022-11-23T02:21:47.9955835Z OK
2022-11-23T02:21:47.9956060Z 
2022-11-23T02:21:47.9956284Z Generating XML reports...
2022-11-23T02:21:47.9957482Z Generated XML report: test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20221123022146.xml
2022-11-23T02:21:47.9958164Z 
2022-11-23T02:21:47.9958807Z ##[endgroup]
2022-11-23T02:21:47.9959421Z FINISHED PRINTING LOG FILE of distributed/test_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_nccl_i7wadmwq)
2022-11-23T02:21:47.9959745Z 
2022-11-23T02:21:47.9960035Z Running distributed/elastic/metrics/api_test ... [2022-11-23 02:21:47.994821]
2022-11-23T02:21:47.9960725Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:21:47.995106]
2022-11-23T02:21:51.9469723Z 
2022-11-23T02:21:51.9470230Z Expand the folded group to see the log file of distributed/elastic/metrics/api_test
2022-11-23T02:21:51.9471460Z ##[group]PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-metrics-api_test_zk2qstse)
2022-11-23T02:21:51.9471869Z 
2022-11-23T02:21:51.9471983Z Running tests...
2022-11-23T02:21:51.9472506Z ----------------------------------------------------------------------
2022-11-23T02:21:51.9473068Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test
2022-11-23T02:21:51.9473783Z   test_get_metric_name (__main__.MetricsApiTest) ... ok (1.643s)
2022-11-23T02:21:51.9474176Z   test_inheritance (__main__.MetricsApiTest) ... ok (0.002s)
2022-11-23T02:21:51.9474514Z   test_profile (__main__.MetricsApiTest) ... ok (0.002s)
2022-11-23T02:21:51.9474720Z 
2022-11-23T02:21:51.9474991Z ----------------------------------------------------------------------
2022-11-23T02:21:51.9475323Z Ran 3 tests in 1.647s
2022-11-23T02:21:51.9475487Z 
2022-11-23T02:21:51.9475582Z OK
2022-11-23T02:21:51.9475700Z 
2022-11-23T02:21:51.9475825Z Generating XML reports...
2022-11-23T02:21:51.9476439Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20221123022149.xml
2022-11-23T02:21:51.9476801Z 
2022-11-23T02:21:51.9477110Z ##[endgroup]
2022-11-23T02:21:51.9477723Z FINISHED PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-metrics-api_test_zk2qstse)
2022-11-23T02:21:51.9478097Z 
2022-11-23T02:21:51.9478377Z Running distributed/nn/jit/test_instantiator ... [2022-11-23 02:21:51.947224]
2022-11-23T02:21:51.9479065Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:21:51.947499]
2022-11-23T02:21:55.8189956Z 
2022-11-23T02:21:55.8190388Z Expand the folded group to see the log file of distributed/nn/jit/test_instantiator
2022-11-23T02:21:55.8191593Z ##[group]PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/workspace/test/test-reports/distributed-nn-jit-test_instantiator_slqvrbv1)
2022-11-23T02:21:55.8192220Z 
2022-11-23T02:21:55.8192334Z Running tests...
2022-11-23T02:21:55.8192835Z ----------------------------------------------------------------------
2022-11-23T02:21:55.8193383Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator
2022-11-23T02:21:55.8193873Z   test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (1.607s)
2022-11-23T02:21:55.8194347Z   test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s)
2022-11-23T02:21:55.8194821Z   test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.014s)
2022-11-23T02:21:55.8195067Z 
2022-11-23T02:21:55.8195338Z ----------------------------------------------------------------------
2022-11-23T02:21:55.8195668Z Ran 3 tests in 1.624s
2022-11-23T02:21:55.8195830Z 
2022-11-23T02:21:55.8195982Z OK
2022-11-23T02:21:55.8196205Z 
2022-11-23T02:21:55.8196415Z Generating XML reports...
2022-11-23T02:21:55.8197087Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20221123022153.xml
2022-11-23T02:21:55.8197450Z 
2022-11-23T02:21:55.8197764Z ##[endgroup]
2022-11-23T02:21:55.8198363Z FINISHED PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/workspace/test/test-reports/distributed-nn-jit-test_instantiator_slqvrbv1)
2022-11-23T02:21:55.8198725Z 
2022-11-23T02:21:55.8199004Z Running distributed/elastic/utils/logging_test ... [2022-11-23 02:21:55.819102]
2022-11-23T02:21:55.8199702Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:21:55.819403]
2022-11-23T02:21:59.8454036Z 
2022-11-23T02:21:59.8454468Z Expand the folded group to see the log file of distributed/elastic/utils/logging_test
2022-11-23T02:21:59.8455465Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_mld4erhq)
2022-11-23T02:21:59.8455866Z 
2022-11-23T02:21:59.8455979Z Running tests...
2022-11-23T02:21:59.8456538Z ----------------------------------------------------------------------
2022-11-23T02:21:59.8457104Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test
2022-11-23T02:21:59.8457793Z   test_derive_module_name (__main__.LoggingTest) ... ok (1.667s)
2022-11-23T02:21:59.8458174Z   test_logger_name (__main__.LoggingTest) ... ok (0.002s)
2022-11-23T02:21:59.8458378Z 
2022-11-23T02:21:59.8458672Z ----------------------------------------------------------------------
2022-11-23T02:21:59.8458987Z Ran 2 tests in 1.670s
2022-11-23T02:21:59.8459149Z 
2022-11-23T02:21:59.8459241Z OK
2022-11-23T02:21:59.8459373Z 
2022-11-23T02:21:59.8459499Z Generating XML reports...
2022-11-23T02:21:59.8460092Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20221123022157.xml
2022-11-23T02:21:59.8460445Z 
2022-11-23T02:21:59.8460755Z ##[endgroup]
2022-11-23T02:21:59.8461379Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_mld4erhq)
2022-11-23T02:21:59.8461755Z 
2022-11-23T02:21:59.8462027Z Running distributed/fsdp/test_checkpoint_wrapper ... [2022-11-23 02:21:59.845430]
2022-11-23T02:21:59.8462745Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:21:59.845711]
2022-11-23T02:22:04.3447821Z 
2022-11-23T02:22:04.3448852Z Expand the folded group to see the log file of distributed/fsdp/test_checkpoint_wrapper
2022-11-23T02:22:04.3450285Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_4qrhoizm)
2022-11-23T02:22:04.3450661Z 
2022-11-23T02:22:04.3450775Z Running tests...
2022-11-23T02:22:04.3451568Z ----------------------------------------------------------------------
2022-11-23T02:22:04.3452147Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper
2022-11-23T02:22:04.3452618Z   test_apply_activation_checkpointing (__main__.CheckpointWrapperTest)
2022-11-23T02:22:04.3453058Z Ensures that `apply_activation_checkpointing` can be used ... ok (1.757s)
2022-11-23T02:22:04.3453508Z   test_checkpoint_wrapper_cpu_offload (__main__.CheckpointWrapperTest) ... ok (0.411s)
2022-11-23T02:22:04.3453981Z   test_checkpoint_wrapper_kwarg_support (__main__.CheckpointWrapperTest) ... ok (0.009s)
2022-11-23T02:22:04.3454398Z   test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest)
2022-11-23T02:22:04.3455508Z Tests that using checkpoint_wrapper or the functional ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79510 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2022-11-23T02:22:04.3456348Z   test_forward_missing_attributes (__main__.CheckpointWrapperTest) ... ok (0.001s)
2022-11-23T02:22:04.3456761Z   test_fqn (__main__.CheckpointWrapperTest) ... ok (0.001s)
2022-11-23T02:22:04.3457176Z   test_load_activation_checkpointed_module (__main__.CheckpointWrapperTest) ... ok (0.003s)
2022-11-23T02:22:04.3457448Z 
2022-11-23T02:22:04.3457719Z ----------------------------------------------------------------------
2022-11-23T02:22:04.3458072Z Ran 7 tests in 2.186s
2022-11-23T02:22:04.3458233Z 
2022-11-23T02:22:04.3458342Z OK (skipped=1)
2022-11-23T02:22:04.3458494Z 
2022-11-23T02:22:04.3458600Z Generating XML reports...
2022-11-23T02:22:04.3459232Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20221123022201.xml
2022-11-23T02:22:04.3459613Z 
2022-11-23T02:22:04.3459924Z ##[endgroup]
2022-11-23T02:22:04.3460548Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_4qrhoizm)
2022-11-23T02:22:04.3460921Z 
2022-11-23T02:22:04.3461241Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2022-11-23 02:22:04.344789]
2022-11-23T02:22:04.3462115Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:22:04.345054]
2022-11-23T02:22:10.7009866Z 
2022-11-23T02:22:10.7010693Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_megatron_prototype
2022-11-23T02:22:10.7011710Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_vwc4tnb6)
2022-11-23T02:22:10.7012131Z 
2022-11-23T02:22:10.7012223Z Running tests...
2022-11-23T02:22:10.7012777Z ----------------------------------------------------------------------
2022-11-23T02:22:10.7013385Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype
2022-11-23T02:22:10.7013973Z   test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:22:10.7014483Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52958
2022-11-23T02:22:10.7014932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52959
2022-11-23T02:22:10.7015374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52960
2022-11-23T02:22:10.7015796Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52961
2022-11-23T02:22:10.7016661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.7017382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.7017978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.7018430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.7019012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.7019456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.7020028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.7020477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.7021049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.7021488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.7022044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.7022510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.7023085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:10.7023525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:10.7024075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:10.7024536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:10.7024968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:10.7025426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:22:10.7025897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:10.7026355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:22:10.7026744Z skip: Need at least 4 CUDA devices (4.042s)
2022-11-23T02:22:10.7027156Z 
2022-11-23T02:22:10.7027538Z ----------------------------------------------------------------------
2022-11-23T02:22:10.7027876Z Ran 1 test in 4.042s
2022-11-23T02:22:10.7028037Z 
2022-11-23T02:22:10.7028142Z OK (skipped=1)
2022-11-23T02:22:10.7028296Z 
2022-11-23T02:22:10.7028418Z Generating XML reports...
2022-11-23T02:22:10.7029107Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20221123022206.xml
2022-11-23T02:22:10.7029535Z 
2022-11-23T02:22:10.7029851Z ##[endgroup]
2022-11-23T02:22:10.7030538Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_vwc4tnb6)
2022-11-23T02:22:10.7030959Z 
2022-11-23T02:22:10.7031265Z Running distributed/_tensor/parallel/test_view_sharding_dim_change ... [2022-11-23 02:22:10.701033]
2022-11-23T02:22:10.7032028Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_view_sharding_dim_change.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:22:10.701442]
2022-11-23T02:22:18.6878896Z 
2022-11-23T02:22:18.6879405Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_view_sharding_dim_change
2022-11-23T02:22:18.6880755Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_view_sharding_dim_change_6tqhv0u2)
2022-11-23T02:22:18.6881202Z 
2022-11-23T02:22:18.6881300Z Running tests...
2022-11-23T02:22:18.6881824Z ----------------------------------------------------------------------
2022-11-23T02:22:18.6882758Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_view_sharding_dim_change
2022-11-23T02:22:18.6883651Z   test_view_with_sharding_dim_change (__main__.TPViewShardingDimChangeTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:22:18.6884169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53129
2022-11-23T02:22:18.6884631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53130
2022-11-23T02:22:18.6885269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:18.6885704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:18.6886279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:18.6886748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:18.6887333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:18.6887757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:18.6888356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:18.6888828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:18.6889268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:18.6889722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:18.6890206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:18.6890695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:18.6891335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:18.6902734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:18.6903135Z ok (5.618s)
2022-11-23T02:22:18.6903275Z 
2022-11-23T02:22:18.6903701Z ----------------------------------------------------------------------
2022-11-23T02:22:18.6904037Z Ran 1 test in 5.619s
2022-11-23T02:22:18.6904180Z 
2022-11-23T02:22:18.6904275Z OK
2022-11-23T02:22:18.6904408Z 
2022-11-23T02:22:18.6904527Z Generating XML reports...
2022-11-23T02:22:18.6905227Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_view_sharding_dim_change/TEST-TPViewShardingDimChangeTest-20221123022212.xml
2022-11-23T02:22:18.6905641Z 
2022-11-23T02:22:18.6905958Z ##[endgroup]
2022-11-23T02:22:18.6906664Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_view_sharding_dim_change_6tqhv0u2)
2022-11-23T02:22:18.6907416Z 
2022-11-23T02:22:18.6907722Z Running distributed/elastic/timer/local_timer_test ... [2022-11-23 02:22:18.687940]
2022-11-23T02:22:18.6908444Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:22:18.688223]
2022-11-23T02:22:27.0670361Z 
2022-11-23T02:22:27.0671111Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_test
2022-11-23T02:22:27.0672300Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_test_pfpjhr7h)
2022-11-23T02:22:27.0672709Z 
2022-11-23T02:22:27.0672834Z Running tests...
2022-11-23T02:22:27.0673369Z ----------------------------------------------------------------------
2022-11-23T02:22:27.0673960Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_test
2022-11-23T02:22:27.0674923Z   test_acquire_release (__main__.LocalTimerServerTest)
2022-11-23T02:22:27.0675957Z tests that: ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/87154 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.649s)
2022-11-23T02:22:27.0676646Z   test_expired_timers (__main__.LocalTimerServerTest)
2022-11-23T02:22:27.0677043Z tests that a single expired timer on a process should terminate ... ok (0.004s)
2022-11-23T02:22:27.0677409Z   test_valid_timers (__main__.LocalTimerServerTest)
2022-11-23T02:22:27.0677825Z tests that valid timers are processed correctly and the process is left alone ... ok (0.003s)
2022-11-23T02:22:27.0678241Z   test_watchdog_call_count (__main__.LocalTimerServerTest)
2022-11-23T02:22:27.0678719Z checks that the watchdog function ran wait/interval +- 1 times ... ok (0.104s)
2022-11-23T02:22:27.0679122Z   test_watchdog_empty_queue (__main__.LocalTimerServerTest)
2022-11-23T02:22:27.0679503Z checks that the watchdog can run on an empty queue ... ok (0.011s)
2022-11-23T02:22:27.0679887Z   test_client_interaction (__main__.LocalTimerTest) ... ok (0.004s)
2022-11-23T02:22:27.0680267Z   test_exception_propagation (__main__.LocalTimerTest) ... ok (0.012s)
2022-11-23T02:22:27.0680639Z   test_get_timer_recursive (__main__.LocalTimerTest)
2022-11-23T02:22:27.0681323Z If a function acquires a countdown timer with default scope, ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:27.0681827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:27.0682427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:27.0682904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:27.0683216Z ok (2.380s)
2022-11-23T02:22:27.0683496Z   test_happy_path (__main__.LocalTimerTest) ... ok (0.103s)
2022-11-23T02:22:27.0683848Z   test_no_client (__main__.LocalTimerTest) ... ok (0.012s)
2022-11-23T02:22:27.0684187Z   test_timer (__main__.LocalTimerTest) ... ok (0.162s)
2022-11-23T02:22:27.0684680Z   test_get (__main__.MultiprocessingRequestQueueTest) ... ok (0.023s)
2022-11-23T02:22:27.0685131Z   test_get_less_than_size (__main__.MultiprocessingRequestQueueTest)
2022-11-23T02:22:27.0685491Z Tests slow producer. ... ok (0.517s)
2022-11-23T02:22:27.0685822Z   test_get_size (__main__.MultiprocessingRequestQueueTest)
2022-11-23T02:22:27.0686230Z Creates a "producer" process that enqueues ``n`` elements ... ok (0.928s)
2022-11-23T02:22:27.0686469Z 
2022-11-23T02:22:27.0686745Z ----------------------------------------------------------------------
2022-11-23T02:22:27.0687077Z Ran 14 tests in 5.918s
2022-11-23T02:22:27.0687246Z 
2022-11-23T02:22:27.0687337Z OK (skipped=1)
2022-11-23T02:22:27.0687495Z 
2022-11-23T02:22:27.0687617Z Generating XML reports...
2022-11-23T02:22:27.0688254Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20221123022220.xml
2022-11-23T02:22:27.0689064Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20221123022220.xml
2022-11-23T02:22:27.0689899Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20221123022220.xml
2022-11-23T02:22:27.0690313Z 
2022-11-23T02:22:27.0690634Z ##[endgroup]
2022-11-23T02:22:27.0691276Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_test_pfpjhr7h)
2022-11-23T02:22:27.0691657Z 
2022-11-23T02:22:27.0691934Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2022-11-23 02:22:27.067115]
2022-11-23T02:22:27.0692739Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:22:27.067378]
2022-11-23T02:22:35.4956075Z 
2022-11-23T02:22:35.4956642Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_wrapping
2022-11-23T02:22:35.4957980Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_q5ogntn9)
2022-11-23T02:22:35.4958446Z 
2022-11-23T02:22:35.4958550Z Running tests...
2022-11-23T02:22:35.4959068Z ----------------------------------------------------------------------
2022-11-23T02:22:35.4959638Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2022-11-23T02:22:35.4960112Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2022-11-23T02:22:35.4960598Z This test simulates wrapping the module after training to run inference. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:22:35.4961101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53332
2022-11-23T02:22:35.4961534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53333
2022-11-23T02:22:35.4962168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:35.4962628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:35.4963184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:35.4963656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:35.4964234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:35.4964687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:35.4965240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:35.4965704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:35.4966389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:35.4966895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:35.4967563Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:35.4968253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:35.4968797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:35.4969277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:35.4970531Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:35.4971320Z   warnings.warn(
2022-11-23T02:22:35.4972473Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:22:35.4973365Z   warnings.warn(
2022-11-23T02:22:35.4973615Z dist init r=1, world=2
2022-11-23T02:22:35.4973846Z dist init r=0, world=2
2022-11-23T02:22:35.4974083Z ok (6.080s)
2022-11-23T02:22:35.4974230Z 
2022-11-23T02:22:35.4974511Z ----------------------------------------------------------------------
2022-11-23T02:22:35.4974822Z Ran 1 test in 6.080s
2022-11-23T02:22:35.4974985Z 
2022-11-23T02:22:35.4975077Z OK
2022-11-23T02:22:35.4975209Z 
2022-11-23T02:22:35.4975332Z Generating XML reports...
2022-11-23T02:22:35.4975968Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221123022229.xml
2022-11-23T02:22:35.4976333Z 
2022-11-23T02:22:35.4976656Z ##[endgroup]
2022-11-23T02:22:35.4977302Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_q5ogntn9)
2022-11-23T02:22:35.4977691Z 
2022-11-23T02:22:35.4978020Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2022-11-23 02:22:35.495736]
2022-11-23T02:22:35.4978772Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:22:35.496028]
2022-11-23T02:22:44.2520319Z 
2022-11-23T02:22:44.2521112Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2022-11-23T02:22:44.2522168Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_6bkcpc4l)
2022-11-23T02:22:44.2522577Z 
2022-11-23T02:22:44.2522698Z Running tests...
2022-11-23T02:22:44.2523239Z ----------------------------------------------------------------------
2022-11-23T02:22:44.2523889Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard
2022-11-23T02:22:44.2524419Z   test_sharded_tensor_reshard (__main__.TestReshard) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:22:44.2524872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53450
2022-11-23T02:22:44.2525568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53451
2022-11-23T02:22:44.2526039Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53452
2022-11-23T02:22:44.2526451Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53453
2022-11-23T02:22:44.2527084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:44.2527540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:44.2528122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:44.2528578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:44.2529160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:44.2529612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:44.2530171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:44.2530638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:44.2531216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:44.2531685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:44.2532260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:44.2532826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:44.2533409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:44.2533856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:44.2534415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:44.2534882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:44.2535316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:44.2535790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:22:44.2536235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:22:44.2536693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:44.2537089Z skip: Need at least 4 CUDA devices (4.046s)
2022-11-23T02:22:44.2537544Z   test_sharded_tensor_reshard_errors (__main__.TestReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53586
2022-11-23T02:22:44.2538066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53587
2022-11-23T02:22:44.2538510Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53588
2022-11-23T02:22:44.2538948Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53589
2022-11-23T02:22:44.2539543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:44.2539992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:44.2540570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:44.2541032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:44.2541618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:44.2542061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:44.2542708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:44.2543155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:44.2543741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:44.2544183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:44.2544757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:44.2545209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:44.2545784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:44.2546226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:44.2547149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:44.2547753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:44.2548185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:44.2548662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:22:44.2549113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:22:44.2549570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:44.2550090Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:22:44.2550288Z 
2022-11-23T02:22:44.2550555Z ----------------------------------------------------------------------
2022-11-23T02:22:44.2550887Z Ran 2 tests in 6.456s
2022-11-23T02:22:44.2551053Z 
2022-11-23T02:22:44.2551164Z OK (skipped=2)
2022-11-23T02:22:44.2551332Z 
2022-11-23T02:22:44.2551448Z Generating XML reports...
2022-11-23T02:22:44.2552055Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20221123022237.xml
2022-11-23T02:22:44.2552418Z 
2022-11-23T02:22:44.2552749Z ##[endgroup]
2022-11-23T02:22:44.2553463Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_6bkcpc4l)
2022-11-23T02:22:44.2553882Z 
2022-11-23T02:22:44.2554162Z Running distributed/_tensor/parallel/test_2d_parallel ... [2022-11-23 02:22:44.252105]
2022-11-23T02:22:44.2554896Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_2d_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:22:44.252388]
2022-11-23T02:22:53.0796227Z 
2022-11-23T02:22:53.0797222Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_2d_parallel
2022-11-23T02:22:53.0798355Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_2d_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_2d_parallel_i0wwavov)
2022-11-23T02:22:53.0798864Z 
2022-11-23T02:22:53.0798977Z Running tests...
2022-11-23T02:22:53.0799586Z ----------------------------------------------------------------------
2022-11-23T02:22:53.0800399Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel
2022-11-23T02:22:53.0801117Z   test_2d_fsdp_integration_correctness (__main__.Test2dParallelIntegration) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:22:53.0801725Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53757
2022-11-23T02:22:53.0802321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53758
2022-11-23T02:22:53.0803322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:53.0803832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:53.0804398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:53.0804870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:53.0805441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:53.0805879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:53.0806433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:53.0806897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:53.0807332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:53.0807807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:53.0808278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:53.0808776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:53.0809432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:53.0810107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:53.0810665Z skip: Need at least 4 CUDA devices (4.005s)
2022-11-23T02:22:53.0811178Z   test_2d_fsdp_integration_functionality (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53827
2022-11-23T02:22:53.0811734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53828
2022-11-23T02:22:53.0812332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:53.0812776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:53.0813350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:53.0813825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:53.0814374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:22:53.0814819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:22:53.0815384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:22:53.0815828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:22:53.0816263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:22:53.0816753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:22:53.0817237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:22:53.0817699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:22:53.0818354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:53.0819040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:22:53.0819483Z skip: Need at least 4 CUDA devices (2.409s)
2022-11-23T02:22:53.0819661Z 
2022-11-23T02:22:53.0819927Z ----------------------------------------------------------------------
2022-11-23T02:22:53.0820257Z Ran 2 tests in 6.415s
2022-11-23T02:22:53.0820418Z 
2022-11-23T02:22:53.0820589Z OK (skipped=2)
2022-11-23T02:22:53.0820752Z 
2022-11-23T02:22:53.0820859Z Generating XML reports...
2022-11-23T02:22:53.0821512Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel/TEST-Test2dParallelIntegration-20221123022246.xml
2022-11-23T02:22:53.0821902Z 
2022-11-23T02:22:53.0822230Z ##[endgroup]
2022-11-23T02:22:53.0822854Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_2d_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_2d_parallel_i0wwavov)
2022-11-23T02:22:53.0823240Z 
2022-11-23T02:22:53.0823513Z Running distributed/_tensor/test_view_ops ... [2022-11-23 02:22:53.079648]
2022-11-23T02:22:53.0824184Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:22:53.079921]
2022-11-23T02:23:02.0951321Z 
2022-11-23T02:23:02.0951923Z Expand the folded group to see the log file of distributed/_tensor/test_view_ops
2022-11-23T02:23:02.0952854Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_view_ops_crn844kw)
2022-11-23T02:23:02.0953259Z 
2022-11-23T02:23:02.0953355Z Running tests...
2022-11-23T02:23:02.0953891Z ----------------------------------------------------------------------
2022-11-23T02:23:02.0954633Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_view_ops
2022-11-23T02:23:02.0955292Z   test_view_groups (__main__.TestViewOps) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:23:02.0956065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53932
2022-11-23T02:23:02.0956514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53933
2022-11-23T02:23:02.0957015Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53934
2022-11-23T02:23:02.0957682Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53935
2022-11-23T02:23:02.0958134Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 53936
2022-11-23T02:23:02.0958582Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 53937
2022-11-23T02:23:02.0959213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:02.0959649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:02.0960228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:02.0960704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:02.0961264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:02.0961712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:02.0962282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:02.0962747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:02.0963304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:02.0963746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:02.0964320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:02.0964767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:02.0965347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:02.0965786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:02.0966477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:02.0966942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:02.0967529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:02.0967972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:02.0968544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:02.0968987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:02.0969569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:02.0970013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:02.0970619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:02.0971063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:02.0971503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:23:02.0971975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:23:02.0972419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:02.0972889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:02.0973348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:02.0973882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:02.0974205Z ok (4.114s)
2022-11-23T02:23:02.0974608Z   test_view_ops (__main__.TestViewOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54136
2022-11-23T02:23:02.0975107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54137
2022-11-23T02:23:02.0975540Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54138
2022-11-23T02:23:02.0975982Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54139
2022-11-23T02:23:02.0976413Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 54140
2022-11-23T02:23:02.0976849Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 54141
2022-11-23T02:23:02.0977452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:02.0977909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:02.0978487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:02.0978941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:02.0979521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:02.0979959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:02.0980529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:02.0980973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:02.0981550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:02.0981994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:02.0982563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:02.0983005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:02.0983653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:02.0984109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:02.0984666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:02.0985132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:02.0985705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:02.0986146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:02.0986697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:02.0987403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:02.0987993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:02.0988413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:02.0988983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:02.0989441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:02.0989875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:23:02.0990330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:02.0990905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:02.0991371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:02.0991842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:23:02.0992292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:02.0992674Z skip: Need at least 6 CUDA devices (2.521s)
2022-11-23T02:23:02.0992867Z 
2022-11-23T02:23:02.0993147Z ----------------------------------------------------------------------
2022-11-23T02:23:02.0993460Z Ran 2 tests in 6.635s
2022-11-23T02:23:02.0993624Z 
2022-11-23T02:23:02.0993733Z OK (skipped=1)
2022-11-23T02:23:02.0993888Z 
2022-11-23T02:23:02.0994014Z Generating XML reports...
2022-11-23T02:23:02.0994578Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_view_ops/TEST-TestViewOps-20221123022255.xml
2022-11-23T02:23:02.0994895Z 
2022-11-23T02:23:02.0995230Z ##[endgroup]
2022-11-23T02:23:02.0995826Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_view_ops_crn844kw)
2022-11-23T02:23:02.0996169Z 
2022-11-23T02:23:02.0996474Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2022-11-23 02:23:02.095233]
2022-11-23T02:23:02.0997185Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:23:02.095510]
2022-11-23T02:23:13.1552750Z 
2022-11-23T02:23:13.1553392Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_linear
2022-11-23T02:23:13.1555003Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_et3pa8ee)
2022-11-23T02:23:13.1555959Z 
2022-11-23T02:23:13.1556172Z Running tests...
2022-11-23T02:23:13.1556746Z ----------------------------------------------------------------------
2022-11-23T02:23:13.1557326Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear
2022-11-23T02:23:13.1558149Z   test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:23:13.1558687Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54375
2022-11-23T02:23:13.1559354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54376
2022-11-23T02:23:13.1560291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54377
2022-11-23T02:23:13.1561164Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54378
2022-11-23T02:23:13.1562489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:13.1563445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:13.1564779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:13.1565274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:13.1565861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:13.1566315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:13.1566872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:13.1567346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:13.1567920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:13.1568514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:13.1569072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:13.1569538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:13.1570118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:13.1570541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:13.1571113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:13.1571573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:13.1572010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:13.1572469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:13.1572932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:13.1573404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:13.1573780Z skip: Need at least 4 CUDA devices (4.044s)
2022-11-23T02:23:13.1574289Z   test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54511
2022-11-23T02:23:13.1574835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54512
2022-11-23T02:23:13.1575281Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54513
2022-11-23T02:23:13.1575703Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54514
2022-11-23T02:23:13.1576305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:13.1576757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:13.1577322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:13.1577788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:13.1578440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:13.1578893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:13.1579450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:13.1579917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:13.1580488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:13.1580927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:13.1581485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:13.1581948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:13.1582521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:13.1582944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:13.1583513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:13.1583974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:13.1584410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:13.1584868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:13.1585328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:13.1585859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:13.1586234Z skip: Need at least 4 CUDA devices (2.412s)
2022-11-23T02:23:13.1586738Z   test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54647
2022-11-23T02:23:13.1587709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54648
2022-11-23T02:23:13.1588158Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54649
2022-11-23T02:23:13.1588582Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54650
2022-11-23T02:23:13.1589203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:13.1589651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:13.1590240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:13.1590691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:13.1591271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:13.1591718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:13.1592261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:13.1592710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:13.1593284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:13.1593753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:13.1594323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:13.1594792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:13.1595371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:13.1595908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:13.1596502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:13.1596966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:13.1597401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:13.1597857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:13.1598318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:13.1598793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:13.1599186Z skip: Need at least 4 CUDA devices (2.310s)
2022-11-23T02:23:13.1599364Z 
2022-11-23T02:23:13.1599638Z ----------------------------------------------------------------------
2022-11-23T02:23:13.1599971Z Ran 3 tests in 8.766s
2022-11-23T02:23:13.1600139Z 
2022-11-23T02:23:13.1600250Z OK (skipped=3)
2022-11-23T02:23:13.1600406Z 
2022-11-23T02:23:13.1600510Z Generating XML reports...
2022-11-23T02:23:13.1601171Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20221123022303.xml
2022-11-23T02:23:13.1601570Z 
2022-11-23T02:23:13.1601895Z ##[endgroup]
2022-11-23T02:23:13.1602541Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_et3pa8ee)
2022-11-23T02:23:13.1603033Z 
2022-11-23T02:23:13.1603329Z Running distributed/elastic/timer/local_timer_example ... [2022-11-23 02:23:13.155399]
2022-11-23T02:23:13.1604049Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:23:13.155712]
2022-11-23T02:23:26.4831056Z 
2022-11-23T02:23:26.4831598Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_example
2022-11-23T02:23:26.4832727Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_9mmy2g6t)
2022-11-23T02:23:26.4833129Z 
2022-11-23T02:23:26.4833246Z Running tests...
2022-11-23T02:23:26.4833745Z ----------------------------------------------------------------------
2022-11-23T02:23:26.4834352Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example
2022-11-23T02:23:26.4835021Z   test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2022-11-23 02:23:16,584 driver: init
2022-11-23T02:23:26.4835563Z [INFO] 2022-11-23 02:23:16,651 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-11-23T02:23:26.4836036Z [INFO] 2022-11-23 02:23:16,651 api: Starting watchdog thread...
2022-11-23T02:23:26.4836605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4837061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4837624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4838092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4838672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4839103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4839676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4840116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4840956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4841534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4842139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4842585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4843167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4843619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4844186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4844656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4845239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4845687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4846241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4846715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4847295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4847742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4848412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4848862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4849439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4849892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4850483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4850949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4851527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4851954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4852528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4852996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4853473Z [INFO] 2022-11-23 02:23:18,722 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4853958Z [INFO] 2022-11-23 02:23:18,724 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4854436Z [INFO] 2022-11-23 02:23:18,729 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4854971Z [INFO] 2022-11-23 02:23:18,738 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4855448Z [INFO] 2022-11-23 02:23:18,739 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4855926Z [INFO] 2022-11-23 02:23:18,744 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4856405Z [INFO] 2022-11-23 02:23:18,748 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4856858Z [INFO] 2022-11-23 02:23:18,748 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4857440Z [INFO] 2022-11-23 02:23:19,847 api: Reaping worker_id=[54819]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:23:26.4857950Z [INFO] 2022-11-23 02:23:19,848 api: Successfully reaped worker=[54819]
2022-11-23T02:23:26.4858575Z [INFO] 2022-11-23 02:23:19,848 api: Reaping worker_id=[54825]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:23:26.4859076Z [INFO] 2022-11-23 02:23:19,848 api: Successfully reaped worker=[54825]
2022-11-23T02:23:26.4859636Z [INFO] 2022-11-23 02:23:19,858 api: Reaping worker_id=[54821]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:23:26.4860147Z [INFO] 2022-11-23 02:23:19,859 api: Successfully reaped worker=[54821]
2022-11-23T02:23:26.4860698Z [INFO] 2022-11-23 02:23:19,869 api: Reaping worker_id=[54823]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:23:26.4861184Z [INFO] 2022-11-23 02:23:19,869 api: Successfully reaped worker=[54823]
2022-11-23T02:23:26.4861628Z [INFO] 2022-11-23 02:23:19,880 api: Stopping LocalTimerServer
2022-11-23T02:23:26.4862055Z [INFO] 2022-11-23 02:23:19,880 api: Stopping watchdog thread...
2022-11-23T02:23:26.4862325Z ok (4.860s)
2022-11-23T02:23:26.4862889Z   test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2022-11-23 02:23:19,892 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2022-11-23T02:23:26.4863426Z [INFO] 2022-11-23 02:23:19,892 api: Starting watchdog thread...
2022-11-23T02:23:26.4863987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4864419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4864993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4865462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4866104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4866553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4867422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4867904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4868466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4868918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4869491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4869950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4870504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4870952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4871508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4871937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4872509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4872974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4873554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4873994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4874570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4875027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4875578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4876041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4876717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4877181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4877740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4878207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4878783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4879236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4879785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4880249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4880742Z [INFO] 2022-11-23 02:23:21,853 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4881210Z [INFO] 2022-11-23 02:23:21,866 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4881683Z [INFO] 2022-11-23 02:23:21,874 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4882155Z [INFO] 2022-11-23 02:23:21,874 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4882636Z [INFO] 2022-11-23 02:23:21,878 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4883095Z [INFO] 2022-11-23 02:23:21,944 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4883670Z [INFO] 2022-11-23 02:23:21,952 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4884147Z [INFO] 2022-11-23 02:23:21,952 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4884702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4885152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4885729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4886197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4886759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4887203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4887770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4888197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4888777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4889248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4889833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4890272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4890846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4891307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4891877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4892335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4892907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4893348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4893964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4894440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4895019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4895459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4896009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4896479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4897046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4897465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4898034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4898490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4899054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:26.4899476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:26.4900039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:26.4900499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:26.4901062Z [INFO] 2022-11-23 02:23:24,907 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4901526Z [INFO] 2022-11-23 02:23:24,912 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4901993Z [INFO] 2022-11-23 02:23:24,914 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4902473Z [INFO] 2022-11-23 02:23:24,916 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4902927Z [INFO] 2022-11-23 02:23:24,932 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4903392Z [INFO] 2022-11-23 02:23:24,938 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4903865Z [INFO] 2022-11-23 02:23:24,943 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4904334Z [INFO] 2022-11-23 02:23:24,945 api: Timer client configured to: LocalTimerClient
2022-11-23T02:23:26.4904888Z [INFO] 2022-11-23 02:23:26,033 api: Reaping worker_id=[55370]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:23:26.4905403Z [INFO] 2022-11-23 02:23:26,034 api: Successfully reaped worker=[55370]
2022-11-23T02:23:26.4905956Z [INFO] 2022-11-23 02:23:26,034 api: Reaping worker_id=[55367]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:23:26.4906441Z [INFO] 2022-11-23 02:23:26,034 api: Successfully reaped worker=[55367]
2022-11-23T02:23:26.4907217Z [INFO] 2022-11-23 02:23:26,034 api: Reaping worker_id=[55364]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:23:26.4907733Z [INFO] 2022-11-23 02:23:26,035 api: Successfully reaped worker=[55364]
2022-11-23T02:23:26.4908283Z [INFO] 2022-11-23 02:23:26,035 api: Reaping worker_id=[55368]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:23:26.4908764Z [INFO] 2022-11-23 02:23:26,036 api: Successfully reaped worker=[55368]
2022-11-23T02:23:26.4909310Z [INFO] 2022-11-23 02:23:26,047 api: Reaping worker_id=[55365]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:23:26.4909821Z [INFO] 2022-11-23 02:23:26,047 api: Successfully reaped worker=[55365]
2022-11-23T02:23:26.4910392Z [INFO] 2022-11-23 02:23:26,057 api: Reaping worker_id=[55371]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:23:26.4910875Z [INFO] 2022-11-23 02:23:26,058 api: Successfully reaped worker=[55371]
2022-11-23T02:23:26.4911518Z [INFO] 2022-11-23 02:23:26,058 api: Reaping worker_id=[55366]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:23:26.4912043Z [INFO] 2022-11-23 02:23:26,058 api: Successfully reaped worker=[55366]
2022-11-23T02:23:26.4912583Z [INFO] 2022-11-23 02:23:26,068 api: Reaping worker_id=[55369]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2022-11-23T02:23:26.4913064Z [INFO] 2022-11-23 02:23:26,069 api: Successfully reaped worker=[55369]
2022-11-23T02:23:26.4913496Z [INFO] 2022-11-23 02:23:26,081 api: Stopping LocalTimerServer
2022-11-23T02:23:26.4913930Z [INFO] 2022-11-23 02:23:26,081 api: Stopping watchdog thread...
2022-11-23T02:23:26.4914200Z ok (6.199s)
2022-11-23T02:23:26.4914348Z 
2022-11-23T02:23:26.4914613Z ----------------------------------------------------------------------
2022-11-23T02:23:26.4914949Z Ran 2 tests in 11.060s
2022-11-23T02:23:26.4915112Z 
2022-11-23T02:23:26.4915187Z OK
2022-11-23T02:23:26.4915322Z 
2022-11-23T02:23:26.4915449Z Generating XML reports...
2022-11-23T02:23:26.4916078Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20221123022315.xml
2022-11-23T02:23:26.4916460Z 
2022-11-23T02:23:26.4916766Z ##[endgroup]
2022-11-23T02:23:26.4917425Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_9mmy2g6t)
2022-11-23T02:23:26.4917815Z 
2022-11-23T02:23:26.4918117Z Running distributed/_tensor/parallel/test_parallelize_api ... [2022-11-23 02:23:26.483321]
2022-11-23T02:23:26.4918950Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/parallel/test_parallelize_api.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:23:26.483584]
2022-11-23T02:23:40.0623604Z 
2022-11-23T02:23:40.0624135Z Expand the folded group to see the log file of distributed/_tensor/parallel/test_parallelize_api
2022-11-23T02:23:40.0625365Z ##[group]PRINTING LOG FILE of distributed/_tensor/parallel/test_parallelize_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_parallelize_api_ukcoyo3n)
2022-11-23T02:23:40.0626067Z 
2022-11-23T02:23:40.0626192Z Running tests...
2022-11-23T02:23:40.0626729Z ----------------------------------------------------------------------
2022-11-23T02:23:40.0627652Z Test results will be stored in test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api
2022-11-23T02:23:40.0628191Z   test_creat_1d_device_mesh (__main__.TensorParallelAPITests) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:23:40.0628705Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55671
2022-11-23T02:23:40.0629331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55672
2022-11-23T02:23:40.0630146Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55673
2022-11-23T02:23:40.0630589Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55674
2022-11-23T02:23:40.0631267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0631721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0632298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0632755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0633342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0633799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0634359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0635080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0635697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0636139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0636688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0637162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0637733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0638186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0638742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0639206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0639646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:40.0640106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:40.0640566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:40.0641035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:40.0641426Z skip: Need at least 4 CUDA devices (3.986s)
2022-11-23T02:23:40.0641903Z   test_creat_1d_device_mesh_error (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55807
2022-11-23T02:23:40.0642574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55808
2022-11-23T02:23:40.0643018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55809
2022-11-23T02:23:40.0643439Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55810
2022-11-23T02:23:40.0644062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0644511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0645084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0645533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0646107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0646559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0647131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0647577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0648159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0648600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0649149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0649609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0650182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0650626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0651177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0651635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0652133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:40.0652601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:40.0653066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:40.0653528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:40.0653922Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:23:40.0654438Z   test_parallelize_mlp (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55943
2022-11-23T02:23:40.0654981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55944
2022-11-23T02:23:40.0655426Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55945
2022-11-23T02:23:40.0655851Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55946
2022-11-23T02:23:40.0656476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0656961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0657533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0658000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0658561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0659005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0659651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0660101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0660679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0661121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0661687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0662131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0662702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0663139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0663693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0664154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0664593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:40.0665069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:40.0665521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:40.0665985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:40.0666374Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:23:40.0667072Z   test_parallelize_mlp_error (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56079
2022-11-23T02:23:40.0667610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56080
2022-11-23T02:23:40.0668063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56081
2022-11-23T02:23:40.0668507Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56082
2022-11-23T02:23:40.0669214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0669681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0670257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0670728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0671288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0671733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0672306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0672751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0673322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0673765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0674334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0674778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0675348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:40.0675789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:40.0676354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:40.0676898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:40.0677336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:40.0677810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:40.0678260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:40.0678720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:40.0679109Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:23:40.0679303Z 
2022-11-23T02:23:40.0679580Z ----------------------------------------------------------------------
2022-11-23T02:23:40.0679897Z Ran 4 tests in 11.217s
2022-11-23T02:23:40.0680059Z 
2022-11-23T02:23:40.0680166Z OK (skipped=4)
2022-11-23T02:23:40.0680319Z 
2022-11-23T02:23:40.0680449Z Generating XML reports...
2022-11-23T02:23:40.0681085Z Generated XML report: test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api/TEST-TensorParallelAPITests-20221123022328.xml
2022-11-23T02:23:40.0681471Z 
2022-11-23T02:23:40.0681806Z ##[endgroup]
2022-11-23T02:23:40.0682478Z FINISHED PRINTING LOG FILE of distributed/_tensor/parallel/test_parallelize_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-parallel-test_parallelize_api_ukcoyo3n)
2022-11-23T02:23:40.0682878Z 
2022-11-23T02:23:40.0683188Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2022-11-23 02:23:40.062401]
2022-11-23T02:23:40.0683905Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:23:40.062799]
2022-11-23T02:23:56.0500090Z 
2022-11-23T02:23:56.0501038Z Expand the folded group to see the log file of distributed/_shard/sharding_plan/test_sharding_plan
2022-11-23T02:23:56.0502284Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_n3fpu5pn)
2022-11-23T02:23:56.0502784Z 
2022-11-23T02:23:56.0502983Z Running tests...
2022-11-23T02:23:56.0503798Z ----------------------------------------------------------------------
2022-11-23T02:23:56.0504690Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan
2022-11-23T02:23:56.0505228Z   test_custom_sharding_planner (__main__.TestShardingPlan) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:23:56.0505931Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56250
2022-11-23T02:23:56.0506385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56251
2022-11-23T02:23:56.0507292Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56252
2022-11-23T02:23:56.0507773Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56253
2022-11-23T02:23:56.0508644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0509114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0509944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0510410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0511227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0511682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0512475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0512937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0513924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0514377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0515195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0515655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0516527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0516986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0517791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0518255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0518860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:56.0519389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:56.0519866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:56.0520529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:56.0520921Z skip: Need at least 4 CUDA devices (4.035s)
2022-11-23T02:23:56.0521622Z   test_reshard_to_ddp_sharding_plan (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56386
2022-11-23T02:23:56.0522149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56387
2022-11-23T02:23:56.0522773Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56388
2022-11-23T02:23:56.0523249Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56389
2022-11-23T02:23:56.0524023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0524524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0525314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0525861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0526539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0527127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0527778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0528414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0529027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0529674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0530251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0530987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0531560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0532231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0532812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0533473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0533916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:56.0534660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:56.0535174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:56.0535643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:56.0536233Z skip: Need at least 4 CUDA devices (2.412s)
2022-11-23T02:23:56.0536719Z   test_shard_module_sub_process_group (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56522
2022-11-23T02:23:56.0537454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56523
2022-11-23T02:23:56.0537921Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56524
2022-11-23T02:23:56.0538539Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56525
2022-11-23T02:23:56.0539188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0539780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0540438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0541032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0541710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0542212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0542956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0543420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0544216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0544665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0545477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0546005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0546962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0547600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0548250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0548809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0549382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:56.0549862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:56.0550538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:56.0551002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:56.0551566Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:23:56.0552069Z   test_sharding_plan_errors (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56658
2022-11-23T02:23:56.0552673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56659
2022-11-23T02:23:56.0553249Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56660
2022-11-23T02:23:56.0553693Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56661
2022-11-23T02:23:56.0554538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0555158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0555936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0556418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0557206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0557648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0558456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0558906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0559725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0560174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0560963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0561411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0562218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0562664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0563476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0563930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0564509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:56.0565064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:56.0565528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:56.0566212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:56.0566604Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:23:56.0567194Z   test_sharding_plan_simple_megatron (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56794
2022-11-23T02:23:56.0567945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56795
2022-11-23T02:23:56.0568392Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56796
2022-11-23T02:23:56.0568832Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56797
2022-11-23T02:23:56.0569437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0570131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0570722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0571181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0571745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0572182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0572750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0573211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0573949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0574441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0575105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0575547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0576125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:23:56.0576561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:23:56.0577128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:23:56.0577568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:23:56.0578001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:23:56.0578472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:23:56.0578920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:23:56.0579382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:23:56.0579758Z skip: Need at least 4 CUDA devices (2.413s)
2022-11-23T02:23:56.0579952Z 
2022-11-23T02:23:56.0580228Z ----------------------------------------------------------------------
2022-11-23T02:23:56.0580541Z Ran 5 tests in 13.681s
2022-11-23T02:23:56.0580701Z 
2022-11-23T02:23:56.0580817Z OK (skipped=5)
2022-11-23T02:23:56.0581090Z 
2022-11-23T02:23:56.0581298Z Generating XML reports...
2022-11-23T02:23:56.0581923Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20221123022341.xml
2022-11-23T02:23:56.0582294Z 
2022-11-23T02:23:56.0582622Z ##[endgroup]
2022-11-23T02:23:56.0583289Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_n3fpu5pn)
2022-11-23T02:23:56.0583694Z 
2022-11-23T02:23:56.0583951Z Running distributed/_tensor/test_api ... [2022-11-23 02:23:56.050338]
2022-11-23T02:23:56.0584676Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_api.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:23:56.050714]
2022-11-23T02:24:12.3197464Z 
2022-11-23T02:24:12.3198087Z Expand the folded group to see the log file of distributed/_tensor/test_api
2022-11-23T02:24:12.3199509Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_api_jep_qkzy)
2022-11-23T02:24:12.3200201Z 
2022-11-23T02:24:12.3200320Z Running tests...
2022-11-23T02:24:12.3201168Z ----------------------------------------------------------------------
2022-11-23T02:24:12.3202417Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_api
2022-11-23T02:24:12.3203325Z   test_distribute_module (__main__.DTensorAPITest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:24:12.3203781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56965
2022-11-23T02:24:12.3204231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56966
2022-11-23T02:24:12.3204683Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56967
2022-11-23T02:24:12.3205117Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56968
2022-11-23T02:24:12.3205953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3206416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3206999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3207457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3208307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3208748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3209327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3209778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3210352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3210799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3211355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3211819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3212394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3212832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3213387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3213851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3214290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:24:12.3214747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:24:12.3215218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:24:12.3215675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:24:12.3216065Z skip: Need at least 4 CUDA devices (4.126s)
2022-11-23T02:24:12.3216539Z   test_distribute_module_input_fn_output_fn (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57101
2022-11-23T02:24:12.3217069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57102
2022-11-23T02:24:12.3217513Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57103
2022-11-23T02:24:12.3218062Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57104
2022-11-23T02:24:12.3218676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3219128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3219703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3220154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3220735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3221177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3221744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3222194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3222764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3223202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3223755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3224215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3224784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3225293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3225850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3226314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3226751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:24:12.3227718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:24:12.3228170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:24:12.3228628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:24:12.3229029Z skip: Need at least 4 CUDA devices (2.511s)
2022-11-23T02:24:12.3229474Z   test_distribute_tensor (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57237
2022-11-23T02:24:12.3229991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57238
2022-11-23T02:24:12.3230434Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57239
2022-11-23T02:24:12.3230874Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57240
2022-11-23T02:24:12.3231488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3231943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3232514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3232960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3233537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3233987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3234555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3235000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3235697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3236158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3236720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3237183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3237761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3238213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3238763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3239227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3239665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:24:12.3240139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:24:12.3240588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:24:12.3241045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:24:12.3241434Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:24:12.3241889Z   test_distribute_tensor_errors (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57373
2022-11-23T02:24:12.3242515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57374
2022-11-23T02:24:12.3242961Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57375
2022-11-23T02:24:12.3243444Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57376
2022-11-23T02:24:12.3244050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3244502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3245075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3245542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3246101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3246551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3247114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3247559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3248117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3248584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3249174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3249618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3250192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3250635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3251210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3251660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3252093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:24:12.3252636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:24:12.3253100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:24:12.3253566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:24:12.3253994Z skip: Need at least 4 CUDA devices (2.411s)
2022-11-23T02:24:12.3254480Z   test_distribute_tensor_uneven_sharding (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57509
2022-11-23T02:24:12.3255002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57510
2022-11-23T02:24:12.3255448Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57511
2022-11-23T02:24:12.3255888Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57512
2022-11-23T02:24:12.3256491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3256945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3257513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3257974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3258528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3258972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3259639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3260101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3260661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3261102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3261672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3262113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3262683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:12.3263126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:12.3263693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:12.3264134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:12.3264574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:24:12.3265048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:24:12.3265497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:24:12.3265960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:24:12.3266342Z skip: Need at least 4 CUDA devices (2.410s)
2022-11-23T02:24:12.3266534Z 
2022-11-23T02:24:12.3267110Z ----------------------------------------------------------------------
2022-11-23T02:24:12.3267513Z Ran 5 tests in 13.869s
2022-11-23T02:24:12.3267679Z 
2022-11-23T02:24:12.3267789Z OK (skipped=5)
2022-11-23T02:24:12.3267951Z 
2022-11-23T02:24:12.3268076Z Generating XML reports...
2022-11-23T02:24:12.3268640Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_api/TEST-DTensorAPITest-20221123022358.xml
2022-11-23T02:24:12.3268971Z 
2022-11-23T02:24:12.3269313Z ##[endgroup]
2022-11-23T02:24:12.3269991Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_api_jep_qkzy)
2022-11-23T02:24:12.3270343Z 
2022-11-23T02:24:12.3270624Z Running distributed/fsdp/test_fsdp_overlap ... [2022-11-23 02:24:12.319869]
2022-11-23T02:24:12.3271291Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:24:12.320256]
2022-11-23T02:24:28.4498798Z 
2022-11-23T02:24:28.4499457Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_overlap
2022-11-23T02:24:28.4500520Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_nrfvs4h3)
2022-11-23T02:24:28.4501177Z 
2022-11-23T02:24:28.4501302Z Running tests...
2022-11-23T02:24:28.4501816Z ----------------------------------------------------------------------
2022-11-23T02:24:28.4502667Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap
2022-11-23T02:24:28.4503767Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:24:28.4504908Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57680
2022-11-23T02:24:28.4505796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:28.4506252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:28.4507220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:28.4508409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:28.4509293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:24:28.4510545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:24:28.4511110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:24:28.4512387Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:24:28.4513178Z   warnings.warn(
2022-11-23T02:24:28.4513412Z dist init r=0, world=1
2022-11-23T02:24:28.4513572Z 
2022-11-23T02:24:28.4513668Z rank0:
2022-11-23T02:24:28.4514161Z   e1: {'cpu_iter': 0.0017934798999999836, 'cpu_wait': 3.4589699999987286e-05, 'gpu_compute': 0.06524480013176799, 'gpu_total': 0.7250751972198486}
2022-11-23T02:24:28.4514735Z   e2: {'cpu_iter': 0.00559457330000015, 'cpu_wait': 3.327830000001697e-05, 'gpu_compute': 0.24848000034689904, 'gpu_total': 2.342131233215332}
2022-11-23T02:24:28.4515312Z   e3: {'cpu_iter': 0.0019209432999999442, 'cpu_wait': 0.1872847468999998, 'gpu_compute': 189.67214660644532, 'gpu_total': 189.9610824584961}
2022-11-23T02:24:28.4515880Z   e4: {'cpu_iter': 0.005694486999999882, 'cpu_wait': 0.18455376130000012, 'gpu_compute': 189.723974609375, 'gpu_total': 190.26251983642578}
2022-11-23T02:24:28.4516195Z ok (13.838s)
2022-11-23T02:24:28.4517213Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2022-11-23T02:24:28.4517820Z 
2022-11-23T02:24:28.4518087Z ----------------------------------------------------------------------
2022-11-23T02:24:28.4518537Z Ran 2 tests in 13.839s
2022-11-23T02:24:28.4518717Z 
2022-11-23T02:24:28.4518824Z OK (skipped=1)
2022-11-23T02:24:28.4518976Z 
2022-11-23T02:24:28.4519082Z Generating XML reports...
2022-11-23T02:24:28.4519731Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20221123022414.xml
2022-11-23T02:24:28.4520584Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20221123022414.xml
2022-11-23T02:24:28.4520966Z 
2022-11-23T02:24:28.4521273Z ##[endgroup]
2022-11-23T02:24:28.4521874Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_nrfvs4h3)
2022-11-23T02:24:28.4522229Z 
2022-11-23T02:24:28.4522506Z Running distributed/test_c10d_object_collectives ... [2022-11-23 02:24:28.449892]
2022-11-23T02:24:28.4523204Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:24:28.450160]
2022-11-23T02:24:47.8416577Z 
2022-11-23T02:24:47.8417081Z Expand the folded group to see the log file of distributed/test_c10d_object_collectives
2022-11-23T02:24:47.8418046Z ##[group]PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_object_collectives_ehzkqmnp)
2022-11-23T02:24:47.8418419Z 
2022-11-23T02:24:47.8418527Z Running tests...
2022-11-23T02:24:47.8419415Z ----------------------------------------------------------------------
2022-11-23T02:24:47.8420346Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_object_collectives
2022-11-23T02:24:47.8420857Z   test_all_gather_object (__main__.TestObjectCollectives) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:24:47.8421503Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57757
2022-11-23T02:24:47.8422141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57758
2022-11-23T02:24:47.8422821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:47.8423276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:47.8423860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:47.8424308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:47.8424894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:47.8425335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:47.8425891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:47.8426360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:47.8427069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:24:47.8427576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:24:47.8428045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:24:47.8428529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:24:47.8429197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:24:47.8429895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:24:47.8430273Z ok (5.501s)
2022-11-23T02:24:47.8430839Z   test_broadcast_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57836
2022-11-23T02:24:47.8431390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57837
2022-11-23T02:24:47.8431981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:47.8432429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:47.8432998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:47.8433462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:47.8434027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:47.8434470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:47.8435040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:47.8435487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:47.8435922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:24:47.8436411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:24:47.8436894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:24:47.8437353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:24:47.8438110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:24:47.8438792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:24:47.8439180Z ok (3.911s)
2022-11-23T02:24:47.8439595Z   test_gather_object (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57915
2022-11-23T02:24:47.8440122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57916
2022-11-23T02:24:47.8440727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:47.8441155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:47.8441725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:47.8442193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:47.8442766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:47.8443187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:47.8443761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:47.8444223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:47.8444661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:24:47.8445132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:24:47.8445615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:24:47.8446094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:24:47.8446735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:24:47.8447419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:24:47.8447808Z ok (3.910s)
2022-11-23T02:24:47.8448299Z   test_scatter_object_list (__main__.TestObjectCollectives) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57996
2022-11-23T02:24:47.8448826Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57997
2022-11-23T02:24:47.8449434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:47.8449878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:47.8450429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:47.8450901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:47.8451477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:24:47.8451916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:24:47.8452470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:24:47.8452933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:24:47.8453410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:24:47.8453898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:24:47.8454367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:24:47.8454918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:24:47.8455575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:24:47.8456243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:24:47.8456637Z ok (3.810s)
2022-11-23T02:24:47.8456788Z 
2022-11-23T02:24:47.8457055Z ----------------------------------------------------------------------
2022-11-23T02:24:47.8457383Z Ran 4 tests in 17.133s
2022-11-23T02:24:47.8457526Z 
2022-11-23T02:24:47.8457621Z OK
2022-11-23T02:24:47.8457752Z 
2022-11-23T02:24:47.8457875Z Generating XML reports...
2022-11-23T02:24:47.8458494Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20221123022430.xml
2022-11-23T02:24:47.8458857Z 
2022-11-23T02:24:47.8459160Z ##[endgroup]
2022-11-23T02:24:47.8459780Z FINISHED PRINTING LOG FILE of distributed/test_c10d_object_collectives (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_object_collectives_ehzkqmnp)
2022-11-23T02:24:47.8460143Z 
2022-11-23T02:24:47.8460413Z Running distributed/test_dynamo_distributed ... [2022-11-23 02:24:47.841704]
2022-11-23T02:24:47.8461105Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:24:47.841970]
2022-11-23T02:25:07.1469685Z 
2022-11-23T02:25:07.1470684Z Expand the folded group to see the log file of distributed/test_dynamo_distributed
2022-11-23T02:25:07.1472038Z ##[group]PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/workspace/test/test-reports/distributed-test_dynamo_distributed_gmpwz2lt)
2022-11-23T02:25:07.1472687Z 
2022-11-23T02:25:07.1472868Z Running tests...
2022-11-23T02:25:07.1473734Z ----------------------------------------------------------------------
2022-11-23T02:25:07.1474643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:07.1475700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:25:07.1476794Z Test results will be stored in test-reports/python-unittest/distributed.test_dynamo_distributed
2022-11-23T02:25:07.1477607Z   test_aot_autograd (__main__.TestDistributed)
2022-11-23T02:25:07.1478252Z Explicitly check AotAutograd family of compilers work, ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:07.1478881Z [2022-11-23 02:24:52,634] torch._dynamo.eval_frame: [DEBUG] skipping __init__ /opt/conda/lib/python3.10/contextlib.py
2022-11-23T02:25:07.1479762Z [2022-11-23 02:24:52,634] torch._dynamo.eval_frame: [DEBUG] skipping __enter__ /opt/conda/lib/python3.10/contextlib.py
2022-11-23T02:25:07.1480447Z [2022-11-23 02:24:52,634] torch._dynamo.eval_frame: [DEBUG] skipping __init__ /opt/conda/lib/python3.10/contextlib.py
2022-11-23T02:25:07.1481351Z [2022-11-23 02:24:52,635] torch._dynamo.eval_frame: [DEBUG] skipping __enter__ /opt/conda/lib/python3.10/contextlib.py
2022-11-23T02:25:07.1482134Z [2022-11-23 02:24:52,635] torch._dynamo.eval_frame: [DEBUG] skipping enable_dynamic /opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
2022-11-23T02:25:07.1483013Z [2022-11-23 02:24:52,637] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing opt_fn
2022-11-23T02:25:07.1483772Z [2022-11-23 02:24:52,637] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:434
2022-11-23T02:25:07.1484538Z [2022-11-23 02:24:52,637] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_DEREF ddp_m []
2022-11-23T02:25:07.1485421Z [2022-11-23 02:24:52,637] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs [UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.1486224Z [2022-11-23 02:24:52,637] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [UnspecializedNNModuleVariable(DistributedDataParallel), TensorVariable()]
2022-11-23T02:25:07.1487652Z [2022-11-23 02:24:52,639] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7f98dbcd87c0, file "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1056> 
2022-11-23T02:25:07.1488163Z  1057           0 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1488559Z                2 LOAD_ATTR                1 (autograd)
2022-11-23T02:25:07.1488993Z                4 LOAD_ATTR                2 (profiler)
2022-11-23T02:25:07.1489315Z                6 LOAD_METHOD              3 (record_function)
2022-11-23T02:25:07.1489498Z 
2022-11-23T02:25:07.1489893Z 1058           8 LOAD_CONST               1 ('DistributedDataParallel.forward')
2022-11-23T02:25:07.1490340Z 
2022-11-23T02:25:07.1490503Z 1057          10 CALL_METHOD              1
2022-11-23T02:25:07.1490809Z               12 SETUP_WITH             147 (to 308)
2022-11-23T02:25:07.1491089Z               14 POP_TOP
2022-11-23T02:25:07.1491233Z 
2022-11-23T02:25:07.1491438Z 1060          16 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1491868Z               18 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.1492184Z               20 CALL_METHOD              0
2022-11-23T02:25:07.1492613Z               22 POP_JUMP_IF_FALSE       39 (to 78)
2022-11-23T02:25:07.1492921Z               24 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1493239Z               26 LOAD_ATTR                5 (require_backward_grad_sync)
2022-11-23T02:25:07.1493599Z               28 POP_JUMP_IF_FALSE       39 (to 78)
2022-11-23T02:25:07.1493947Z 
2022-11-23T02:25:07.1494081Z 1061          30 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1494359Z               32 LOAD_ATTR                6 (logger)
2022-11-23T02:25:07.1494653Z               34 LOAD_CONST               0 (None)
2022-11-23T02:25:07.1494951Z               36 IS_OP                    1
2022-11-23T02:25:07.1495219Z               38 POP_JUMP_IF_TRUE        22 (to 44)
2022-11-23T02:25:07.1495502Z               40 LOAD_ASSERTION_ERROR
2022-11-23T02:25:07.1495993Z               42 RAISE_VARARGS            1
2022-11-23T02:25:07.1496182Z 
2022-11-23T02:25:07.1496303Z 1062     >>   44 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1496675Z               46 LOAD_ATTR                6 (logger)
2022-11-23T02:25:07.1497015Z               48 LOAD_METHOD              7 (set_runtime_stats_and_log)
2022-11-23T02:25:07.1497313Z               50 CALL_METHOD              0
2022-11-23T02:25:07.1497565Z               52 POP_TOP
2022-11-23T02:25:07.1497727Z 
2022-11-23T02:25:07.1497850Z 1063          54 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1498238Z               56 DUP_TOP
2022-11-23T02:25:07.1498564Z               58 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.1498858Z               60 LOAD_CONST               2 (1)
2022-11-23T02:25:07.1499126Z               62 INPLACE_ADD
2022-11-23T02:25:07.1499515Z               64 ROT_TWO
2022-11-23T02:25:07.1499859Z               66 STORE_ATTR               8 (num_iterations)
2022-11-23T02:25:07.1500041Z 
2022-11-23T02:25:07.1500168Z 1064          68 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1500543Z               70 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1500937Z               72 LOAD_METHOD             10 (prepare_for_forward)
2022-11-23T02:25:07.1501257Z               74 CALL_METHOD              0
2022-11-23T02:25:07.1501510Z               76 POP_TOP
2022-11-23T02:25:07.1501738Z 
2022-11-23T02:25:07.1501972Z 1068     >>   78 LOAD_GLOBAL             11 (Join)
2022-11-23T02:25:07.1502298Z               80 LOAD_METHOD             12 (notify_join_context)
2022-11-23T02:25:07.1502603Z               82 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1502873Z               84 CALL_METHOD              1
2022-11-23T02:25:07.1503160Z               86 STORE_FAST               3 (work)
2022-11-23T02:25:07.1503408Z 
2022-11-23T02:25:07.1503542Z 1069          88 LOAD_FAST                3 (work)
2022-11-23T02:25:07.1503828Z               90 POP_JUMP_IF_FALSE       54 (to 108)
2022-11-23T02:25:07.1504009Z 
2022-11-23T02:25:07.1504137Z 1070          92 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1504435Z               94 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1504760Z               96 LOAD_METHOD             13 (_set_forward_pass_work_handle)
2022-11-23T02:25:07.1504961Z 
2022-11-23T02:25:07.1505095Z 1071          98 LOAD_FAST                3 (work)
2022-11-23T02:25:07.1505366Z              100 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1505700Z              102 LOAD_ATTR               14 (_divide_by_initial_world_size)
2022-11-23T02:25:07.1505904Z 
2022-11-23T02:25:07.1506015Z 1070         104 CALL_METHOD              2
2022-11-23T02:25:07.1506268Z              106 POP_TOP
2022-11-23T02:25:07.1506426Z 
2022-11-23T02:25:07.1506569Z 1080     >>  108 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1507443Z              110 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.1507821Z              112 CALL_METHOD              0
2022-11-23T02:25:07.1508123Z              114 POP_JUMP_IF_FALSE       71 (to 142)
2022-11-23T02:25:07.1508421Z              116 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1508706Z              118 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1509017Z              120 LOAD_METHOD             15 (_rebuild_buckets)
2022-11-23T02:25:07.1509315Z              122 CALL_METHOD              0
2022-11-23T02:25:07.1509587Z              124 POP_JUMP_IF_FALSE       71 (to 142)
2022-11-23T02:25:07.1509766Z 
2022-11-23T02:25:07.1509906Z 1081         126 LOAD_GLOBAL              6 (logger)
2022-11-23T02:25:07.1510206Z              128 LOAD_METHOD             16 (info)
2022-11-23T02:25:07.1510374Z 
2022-11-23T02:25:07.1510706Z 1082         130 LOAD_CONST               3 ('Reducer buckets have been rebuilt in this iteration.')
2022-11-23T02:25:07.1510965Z 
2022-11-23T02:25:07.1511063Z 1081         132 CALL_METHOD              1
2022-11-23T02:25:07.1511329Z              134 POP_TOP
2022-11-23T02:25:07.1511479Z 
2022-11-23T02:25:07.1511615Z 1084         136 LOAD_CONST               4 (True)
2022-11-23T02:25:07.1511881Z              138 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1512305Z              140 STORE_ATTR              17 (_has_rebuilt_buckets)
2022-11-23T02:25:07.1512521Z 
2022-11-23T02:25:07.1512658Z 1088     >>  142 LOAD_GLOBAL             18 (hasattr)
2022-11-23T02:25:07.1512961Z              144 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1513329Z              146 LOAD_CONST               5 ('buffer_hook')
2022-11-23T02:25:07.1513624Z              148 CALL_FUNCTION            2
2022-11-23T02:25:07.1513942Z              150 STORE_FAST               4 (buffer_hook_registered)
2022-11-23T02:25:07.1514147Z 
2022-11-23T02:25:07.1514260Z 1089         152 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1514582Z              154 LOAD_METHOD             19 (_check_sync_bufs_pre_fwd)
2022-11-23T02:25:07.1514893Z              156 CALL_METHOD              0
2022-11-23T02:25:07.1515185Z              158 POP_JUMP_IF_FALSE       84 (to 168)
2022-11-23T02:25:07.1515356Z 
2022-11-23T02:25:07.1515488Z 1090         160 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1515788Z              162 LOAD_METHOD             20 (_sync_buffers)
2022-11-23T02:25:07.1516086Z              164 CALL_METHOD              0
2022-11-23T02:25:07.1516322Z              166 POP_TOP
2022-11-23T02:25:07.1516477Z 
2022-11-23T02:25:07.1516604Z 1092     >>  168 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1516900Z              170 LOAD_ATTR               21 (_join_config)
2022-11-23T02:25:07.1517177Z              172 LOAD_ATTR               22 (enable)
2022-11-23T02:25:07.1517486Z              174 POP_JUMP_IF_FALSE       94 (to 188)
2022-11-23T02:25:07.1517659Z 
2022-11-23T02:25:07.1517795Z 1094         176 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1518227Z              178 LOAD_ATTR               23 (_check_global_requires_backward_grad_sync)
2022-11-23T02:25:07.1518458Z 
2022-11-23T02:25:07.1518580Z 1095         180 LOAD_CONST               6 (False)
2022-11-23T02:25:07.1518760Z 
2022-11-23T02:25:07.1518998Z 1094         182 LOAD_CONST               7 (('is_joined_rank',))
2022-11-23T02:25:07.1519292Z              184 CALL_FUNCTION_KW         1
2022-11-23T02:25:07.1519601Z              186 POP_TOP
2022-11-23T02:25:07.1519758Z 
2022-11-23T02:25:07.1519887Z 1098     >>  188 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1520197Z              190 LOAD_ATTR               24 (_run_ddp_forward)
2022-11-23T02:25:07.1520491Z              192 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.1520761Z              194 BUILD_MAP                0
2022-11-23T02:25:07.1521042Z              196 LOAD_FAST                2 (kwargs)
2022-11-23T02:25:07.1521330Z              198 DICT_MERGE               1
2022-11-23T02:25:07.1521597Z              200 CALL_FUNCTION_EX         1
2022-11-23T02:25:07.1521873Z              202 STORE_FAST               5 (output)
2022-11-23T02:25:07.1522068Z 
2022-11-23T02:25:07.1522193Z 1102         204 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1522495Z              206 LOAD_METHOD             25 (_check_sync_bufs_post_fwd)
2022-11-23T02:25:07.1522810Z              208 CALL_METHOD              0
2022-11-23T02:25:07.1523098Z              210 POP_JUMP_IF_FALSE      110 (to 220)
2022-11-23T02:25:07.1523296Z 
2022-11-23T02:25:07.1523425Z 1103         212 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1523711Z              214 LOAD_METHOD             20 (_sync_buffers)
2022-11-23T02:25:07.1524000Z              216 CALL_METHOD              0
2022-11-23T02:25:07.1524255Z              218 POP_TOP
2022-11-23T02:25:07.1524416Z 
2022-11-23T02:25:07.1524526Z 1105     >>  220 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1524827Z              222 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.1525137Z              224 CALL_METHOD              0
2022-11-23T02:25:07.1525412Z              226 POP_JUMP_IF_FALSE      144 (to 288)
2022-11-23T02:25:07.1525685Z              228 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1526015Z              230 LOAD_ATTR                5 (require_backward_grad_sync)
2022-11-23T02:25:07.1526388Z              232 POP_JUMP_IF_FALSE      144 (to 288)
2022-11-23T02:25:07.1526577Z 
2022-11-23T02:25:07.1526706Z 1106         234 LOAD_CONST               4 (True)
2022-11-23T02:25:07.1527000Z              236 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1527329Z              238 STORE_ATTR              26 (require_forward_param_sync)
2022-11-23T02:25:07.1527532Z 
2022-11-23T02:25:07.1527642Z 1112         240 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1527963Z              242 LOAD_ATTR               27 (find_unused_parameters)
2022-11-23T02:25:07.1528277Z              244 POP_JUMP_IF_FALSE      137 (to 274)
2022-11-23T02:25:07.1528583Z              246 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1528858Z              248 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.1529178Z              250 POP_JUMP_IF_TRUE       137 (to 274)
2022-11-23T02:25:07.1529350Z 
2022-11-23T02:25:07.1529490Z 1114         252 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1529758Z              254 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1530068Z              256 LOAD_METHOD             29 (prepare_for_backward)
2022-11-23T02:25:07.1530275Z 
2022-11-23T02:25:07.1530405Z 1115         258 LOAD_GLOBAL             30 (list)
2022-11-23T02:25:07.1530707Z              260 LOAD_GLOBAL             31 (_find_tensors)
2022-11-23T02:25:07.1530983Z              262 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1531274Z              264 CALL_FUNCTION            1
2022-11-23T02:25:07.1531548Z              266 CALL_FUNCTION            1
2022-11-23T02:25:07.1531710Z 
2022-11-23T02:25:07.1531837Z 1114         268 CALL_METHOD              1
2022-11-23T02:25:07.1532161Z              270 POP_TOP
2022-11-23T02:25:07.1532433Z              272 JUMP_FORWARD            10 (to 294)
2022-11-23T02:25:07.1532609Z 
2022-11-23T02:25:07.1532729Z 1118     >>  274 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1533013Z              276 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1533339Z              278 LOAD_METHOD             29 (prepare_for_backward)
2022-11-23T02:25:07.1533646Z              280 BUILD_LIST               0
2022-11-23T02:25:07.1533906Z              282 CALL_METHOD              1
2022-11-23T02:25:07.1534151Z              284 POP_TOP
2022-11-23T02:25:07.1534427Z              286 JUMP_FORWARD             3 (to 294)
2022-11-23T02:25:07.1534592Z 
2022-11-23T02:25:07.1534722Z 1120     >>  288 LOAD_CONST               6 (False)
2022-11-23T02:25:07.1535011Z              290 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1535322Z              292 STORE_ATTR              26 (require_forward_param_sync)
2022-11-23T02:25:07.1535618Z          >>  294 POP_BLOCK
2022-11-23T02:25:07.1535779Z 
2022-11-23T02:25:07.1535909Z 1057         296 LOAD_CONST               0 (None)
2022-11-23T02:25:07.1536171Z              298 DUP_TOP
2022-11-23T02:25:07.1536376Z              300 DUP_TOP
2022-11-23T02:25:07.1536646Z              302 CALL_FUNCTION            3
2022-11-23T02:25:07.1536916Z              304 POP_TOP
2022-11-23T02:25:07.1537177Z              306 JUMP_FORWARD             8 (to 324)
2022-11-23T02:25:07.1537435Z          >>  308 WITH_EXCEPT_START
2022-11-23T02:25:07.1537725Z              310 POP_JUMP_IF_TRUE       157 (to 314)
2022-11-23T02:25:07.1538020Z              312 RERAISE                  1
2022-11-23T02:25:07.1538258Z          >>  314 POP_TOP
2022-11-23T02:25:07.1538490Z              316 POP_TOP
2022-11-23T02:25:07.1538720Z              318 POP_TOP
2022-11-23T02:25:07.1538945Z              320 POP_EXCEPT
2022-11-23T02:25:07.1539204Z              322 POP_TOP
2022-11-23T02:25:07.1539351Z 
2022-11-23T02:25:07.1539486Z 1124     >>  324 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1539784Z              326 LOAD_ATTR               27 (find_unused_parameters)
2022-11-23T02:25:07.1540108Z              328 POP_JUMP_IF_FALSE      168 (to 336)
2022-11-23T02:25:07.1540410Z              330 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1540746Z              332 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.1541067Z              334 POP_JUMP_IF_FALSE      178 (to 356)
2022-11-23T02:25:07.1541250Z 
2022-11-23T02:25:07.1541385Z 1125     >>  336 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1541667Z              338 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.1541834Z 
2022-11-23T02:25:07.1541972Z 1124         340 EXTENDED_ARG             1
2022-11-23T02:25:07.1542266Z              342 POP_JUMP_IF_FALSE      265 (to 530)
2022-11-23T02:25:07.1542458Z 
2022-11-23T02:25:07.1542591Z 1125         344 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1542873Z              346 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.1543170Z              348 LOAD_CONST               2 (1)
2022-11-23T02:25:07.1543450Z              350 COMPARE_OP               2 (==)
2022-11-23T02:25:07.1543641Z 
2022-11-23T02:25:07.1543774Z 1124         352 EXTENDED_ARG             1
2022-11-23T02:25:07.1544055Z              354 POP_JUMP_IF_FALSE      265 (to 530)
2022-11-23T02:25:07.1544241Z 
2022-11-23T02:25:07.1544373Z 1128     >>  356 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1544667Z              358 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.1544853Z 
2022-11-23T02:25:07.1544965Z 1129         360 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1545250Z              362 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.1545433Z 
2022-11-23T02:25:07.1545708Z 1127         364 LOAD_CONST               8 (('static_graph', 'num_iterations'))
2022-11-23T02:25:07.1546043Z              366 BUILD_CONST_KEY_MAP      2
2022-11-23T02:25:07.1546390Z              368 STORE_FAST               6 (state_dict)
2022-11-23T02:25:07.1546569Z 
2022-11-23T02:25:07.1546729Z 1136         370 LOAD_GLOBAL             32 (_tree_flatten_with_rref)
2022-11-23T02:25:07.1547449Z              372 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1547724Z              374 CALL_FUNCTION            1
2022-11-23T02:25:07.1547895Z 
2022-11-23T02:25:07.1548031Z 1132         376 UNPACK_SEQUENCE          3
2022-11-23T02:25:07.1548211Z 
2022-11-23T02:25:07.1548367Z 1133         378 STORE_FAST               7 (output_tensor_list)
2022-11-23T02:25:07.1548561Z 
2022-11-23T02:25:07.1548704Z 1134         380 STORE_FAST               8 (treespec)
2022-11-23T02:25:07.1548886Z 
2022-11-23T02:25:07.1549017Z 1135         382 STORE_FAST               9 (output_is_rref)
2022-11-23T02:25:07.1549205Z 
2022-11-23T02:25:07.1549708Z 1137         384 LOAD_CONST               9 (<code object <listcomp> at 0x7f98dbcd8660, file "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1137>)
2022-11-23T02:25:07.1550352Z              386 LOAD_CONST              10 ('DistributedDataParallel.forward.<locals>.<listcomp>')
2022-11-23T02:25:07.1550726Z              388 MAKE_FUNCTION            0
2022-11-23T02:25:07.1550994Z              390 LOAD_GLOBAL             33 (range)
2022-11-23T02:25:07.1551285Z              392 LOAD_GLOBAL             34 (len)
2022-11-23T02:25:07.1551599Z              394 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.1551887Z              396 CALL_FUNCTION            1
2022-11-23T02:25:07.1552165Z              398 CALL_FUNCTION            1
2022-11-23T02:25:07.1552414Z              400 GET_ITER
2022-11-23T02:25:07.1552656Z              402 CALL_FUNCTION            1
2022-11-23T02:25:07.1553008Z              404 STORE_FAST              10 (output_placeholders)
2022-11-23T02:25:07.1553210Z 
2022-11-23T02:25:07.1553353Z 1140         406 LOAD_GLOBAL             35 (enumerate)
2022-11-23T02:25:07.1553668Z              408 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.1553958Z              410 CALL_FUNCTION            1
2022-11-23T02:25:07.1554213Z              412 GET_ITER
2022-11-23T02:25:07.1554476Z          >>  414 FOR_ITER                18 (to 452)
2022-11-23T02:25:07.1554744Z              416 UNPACK_SEQUENCE          2
2022-11-23T02:25:07.1555133Z              418 STORE_FAST              11 (i)
2022-11-23T02:25:07.1555437Z              420 STORE_FAST               5 (output)
2022-11-23T02:25:07.1555613Z 
2022-11-23T02:25:07.1555732Z 1141         422 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1556024Z              424 LOAD_METHOD             36 (is_tensor)
2022-11-23T02:25:07.1556312Z              426 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1556597Z              428 CALL_METHOD              1
2022-11-23T02:25:07.1556874Z              430 POP_JUMP_IF_FALSE      225 (to 450)
2022-11-23T02:25:07.1557170Z              432 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1557459Z              434 LOAD_ATTR               37 (grad_fn)
2022-11-23T02:25:07.1557730Z              436 LOAD_CONST               0 (None)
2022-11-23T02:25:07.1558008Z              438 IS_OP                    0
2022-11-23T02:25:07.1558297Z              440 POP_JUMP_IF_FALSE      225 (to 450)
2022-11-23T02:25:07.1558483Z 
2022-11-23T02:25:07.1558603Z 1142         442 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1558917Z              444 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1559229Z              446 LOAD_FAST               11 (i)
2022-11-23T02:25:07.1559475Z              448 STORE_SUBSCR
2022-11-23T02:25:07.1559754Z          >>  450 JUMP_ABSOLUTE          207 (to 414)
2022-11-23T02:25:07.1559932Z 
2022-11-23T02:25:07.1560071Z 1149     >>  452 LOAD_GLOBAL             38 (_DDPSink)
2022-11-23T02:25:07.1560363Z              454 LOAD_ATTR               39 (apply)
2022-11-23T02:25:07.1560521Z 
2022-11-23T02:25:07.1560652Z 1150         456 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1561027Z              458 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1561200Z 
2022-11-23T02:25:07.1561339Z 1151         460 LOAD_FAST                6 (state_dict)
2022-11-23T02:25:07.1561519Z 
2022-11-23T02:25:07.1561628Z 1149         462 BUILD_LIST               2
2022-11-23T02:25:07.1561796Z 
2022-11-23T02:25:07.1561951Z 1152         464 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.1562145Z 
2022-11-23T02:25:07.1562271Z 1149         466 LIST_EXTEND              1
2022-11-23T02:25:07.1562533Z              468 LIST_TO_TUPLE
2022-11-23T02:25:07.1562789Z              470 CALL_FUNCTION_EX         0
2022-11-23T02:25:07.1563114Z              472 STORE_FAST              12 (passthrough_tensor_list)
2022-11-23T02:25:07.1563322Z 
2022-11-23T02:25:07.1563457Z 1154         474 LOAD_GLOBAL             33 (range)
2022-11-23T02:25:07.1563727Z              476 LOAD_GLOBAL             34 (len)
2022-11-23T02:25:07.1564040Z              478 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1564351Z              480 CALL_FUNCTION            1
2022-11-23T02:25:07.1564627Z              482 CALL_FUNCTION            1
2022-11-23T02:25:07.1564861Z              484 GET_ITER
2022-11-23T02:25:07.1565126Z          >>  486 FOR_ITER                15 (to 518)
2022-11-23T02:25:07.1565410Z              488 STORE_FAST              11 (i)
2022-11-23T02:25:07.1565569Z 
2022-11-23T02:25:07.1565729Z 1155         490 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1566034Z              492 LOAD_FAST               11 (i)
2022-11-23T02:25:07.1566298Z              494 BINARY_SUBSCR
2022-11-23T02:25:07.1566555Z              496 LOAD_CONST               0 (None)
2022-11-23T02:25:07.1566831Z              498 IS_OP                    0
2022-11-23T02:25:07.1567105Z              500 EXTENDED_ARG             1
2022-11-23T02:25:07.1567381Z              502 POP_JUMP_IF_FALSE      258 (to 516)
2022-11-23T02:25:07.1567568Z 
2022-11-23T02:25:07.1567736Z 1156         504 LOAD_FAST               12 (passthrough_tensor_list)
2022-11-23T02:25:07.1568054Z              506 LOAD_FAST               11 (i)
2022-11-23T02:25:07.1568316Z              508 BINARY_SUBSCR
2022-11-23T02:25:07.1568598Z              510 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1568901Z              512 LOAD_FAST               11 (i)
2022-11-23T02:25:07.1569219Z              514 STORE_SUBSCR
2022-11-23T02:25:07.1569492Z          >>  516 JUMP_ABSOLUTE          243 (to 486)
2022-11-23T02:25:07.1569670Z 
2022-11-23T02:25:07.1569835Z 1159     >>  518 LOAD_GLOBAL             40 (_tree_unflatten_with_rref)
2022-11-23T02:25:07.1570039Z 
2022-11-23T02:25:07.1570195Z 1160         520 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1570508Z              522 LOAD_FAST                8 (treespec)
2022-11-23T02:25:07.1570798Z              524 LOAD_FAST                9 (output_is_rref)
2022-11-23T02:25:07.1570985Z 
2022-11-23T02:25:07.1571113Z 1159         526 CALL_FUNCTION            3
2022-11-23T02:25:07.1571407Z              528 STORE_FAST               5 (output)
2022-11-23T02:25:07.1571585Z 
2022-11-23T02:25:07.1571702Z 1162     >>  530 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1571967Z              532 RETURN_VALUE
2022-11-23T02:25:07.1572199Z  
2022-11-23T02:25:07.1572327Z 
2022-11-23T02:25:07.1572788Z [2022-11-23 02:24:52,642] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1057
2022-11-23T02:25:07.1573401Z [2022-11-23 02:24:52,642] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
2022-11-23T02:25:07.1574114Z [2022-11-23 02:24:52,642] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR autograd [TorchVariable(<module 'torch' from '/opt/conda/lib/python3.10/site-packages/torch/__init__.py'>)]
2022-11-23T02:25:07.1574976Z [2022-11-23 02:24:52,643] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR profiler [TorchVariable(<module 'torch.autograd' from '/opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py'>)]
2022-11-23T02:25:07.1576001Z [2022-11-23 02:24:52,643] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR record_function [TorchVariable(<module 'torch.autograd.profiler' from '/opt/conda/lib/python3.10/site-packages/torch/autograd/profiler.py'>)]
2022-11-23T02:25:07.1576815Z [2022-11-23 02:24:52,643] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1058
2022-11-23T02:25:07.1577629Z [2022-11-23 02:24:52,643] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST DistributedDataParallel.forward [TorchVariable(<class 'torch.autograd.profiler.record_function'>)]
2022-11-23T02:25:07.1578434Z [2022-11-23 02:24:52,644] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1057
2022-11-23T02:25:07.1579227Z [2022-11-23 02:24:52,644] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TorchVariable(<class 'torch.autograd.profiler.record_function'>), ConstantVariable(str)]
2022-11-23T02:25:07.1579857Z [2022-11-23 02:24:52,644] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored
2022-11-23T02:25:07.1580425Z [2022-11-23 02:24:52,644] torch._dynamo.symbolic_convert: [DEBUG] TRACE SETUP_WITH 308 [NullContextVariable()]
2022-11-23T02:25:07.1581091Z [2022-11-23 02:24:52,644] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType)]
2022-11-23T02:25:07.1581828Z [2022-11-23 02:24:52,644] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1060
2022-11-23T02:25:07.1582472Z [2022-11-23 02:24:52,644] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [WithExitFunctionVariable()]
2022-11-23T02:25:07.1583302Z [2022-11-23 02:24:52,644] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR is_grad_enabled [WithExitFunctionVariable(), TorchVariable(<module 'torch' from '/opt/conda/lib/python3.10/site-packages/torch/__init__.py'>)]
2022-11-23T02:25:07.1584151Z [2022-11-23 02:24:52,645] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), TorchVariable(<built-in function is_grad_enabled>)]
2022-11-23T02:25:07.1584952Z [2022-11-23 02:24:52,645] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 78 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.1585614Z [2022-11-23 02:24:52,645] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.1586396Z [2022-11-23 02:24:52,645] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR require_backward_grad_sync [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.1587720Z [2022-11-23 02:24:52,645] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 78 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.1588465Z [2022-11-23 02:24:52,646] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1061
2022-11-23T02:25:07.1589129Z [2022-11-23 02:24:52,646] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.1589879Z [2022-11-23 02:24:52,646] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR logger [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.1590671Z [2022-11-23 02:24:52,647] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger)]
2022-11-23T02:25:07.1591458Z [2022-11-23 02:24:52,647] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger), ConstantVariable(NoneType)]
2022-11-23T02:25:07.1592227Z [2022-11-23 02:24:52,647] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 44 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.1593056Z [2022-11-23 02:24:52,647] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1062
2022-11-23T02:25:07.1593717Z [2022-11-23 02:24:52,648] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.1594466Z [2022-11-23 02:24:52,648] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR logger [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.1595288Z [2022-11-23 02:24:52,648] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR set_runtime_stats_and_log [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger)]
2022-11-23T02:25:07.1596065Z [2022-11-23 02:24:52,649] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), UserDefinedObjectVariable(instancemethod)]
2022-11-23T02:25:07.1596823Z [2022-11-23 02:24:52,650] torch._dynamo.convert_frame: [CODE] ORIGINAL BYTECODE opt_fn /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 432 
2022-11-23T02:25:07.1597270Z 434           0 LOAD_DEREF               0 (ddp_m)
2022-11-23T02:25:07.1597564Z               2 LOAD_FAST                0 (inputs)
2022-11-23T02:25:07.1597837Z               4 CALL_FUNCTION            1
2022-11-23T02:25:07.1598152Z               6 RETURN_VALUE
2022-11-23T02:25:07.1598317Z 
2022-11-23T02:25:07.1598407Z  
2022-11-23T02:25:07.1598976Z [2022-11-23 02:24:52,650] torch._dynamo.convert_frame: [CODE] MODIFIED BYTECODE opt_fn /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 432 
2022-11-23T02:25:07.1599409Z 432           0 LOAD_DEREF               0 (ddp_m)
2022-11-23T02:25:07.1599698Z               2 LOAD_FAST                0 (inputs)
2022-11-23T02:25:07.1599873Z 
2022-11-23T02:25:07.1600002Z 434           4 CALL_FUNCTION            1
2022-11-23T02:25:07.1600251Z               6 RETURN_VALUE
2022-11-23T02:25:07.1600415Z 
2022-11-23T02:25:07.1600506Z  
2022-11-23T02:25:07.1600868Z [2022-11-23 02:24:52,783] torch._dynamo.convert_frame: [CODE] GUARDS:
2022-11-23T02:25:07.1601151Z  - 
2022-11-23T02:25:07.1601436Z             local 'ddp_m' TYPE_MATCH
2022-11-23T02:25:07.1601690Z             {
2022-11-23T02:25:07.1602008Z                 'guard_types': ['TYPE_MATCH'],
2022-11-23T02:25:07.1602467Z                 'code': ['___check_type_id(ddp_m, 94163481642224)'],
2022-11-23T02:25:07.1603007Z                 'obj_weakref': <weakref at 0x7f98bc6eb100; to 'DistributedDataParallel' at 0x7f98d16376a0>
2022-11-23T02:25:07.1603615Z                 'guarded_class': <weakref at 0x7f98dbce1a80; to 'ABCMeta' at 0x55a424dfc4f0 (DistributedDataParallel)>
2022-11-23T02:25:07.1603957Z             }
2022-11-23T02:25:07.1604175Z             
2022-11-23T02:25:07.1604400Z  - 
2022-11-23T02:25:07.1604675Z             local 'inputs' TENSOR_MATCH
2022-11-23T02:25:07.1604930Z             {
2022-11-23T02:25:07.1605257Z                 'guard_types': ['TENSOR_MATCH'],
2022-11-23T02:25:07.1605559Z                 'code': None,
2022-11-23T02:25:07.1605985Z                 'obj_weakref': <weakref at 0x7f98bc6eaca0; to 'Tensor' at 0x7f98da2a7d30>
2022-11-23T02:25:07.1606533Z                 'guarded_class': <weakref at 0x7f98dc435d50; to 'torch._C._TensorMeta' at 0x55a4248e47a0 (Tensor)>
2022-11-23T02:25:07.1606864Z             }
2022-11-23T02:25:07.1607064Z             
2022-11-23T02:25:07.1607603Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping _call_impl /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py
2022-11-23T02:25:07.1608308Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping forward /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py
2022-11-23T02:25:07.1608990Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping __init__ /opt/conda/lib/python3.10/site-packages/torch/autograd/profiler.py
2022-11-23T02:25:07.1609681Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping inner /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1610259Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping __getitem__ /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1610840Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping Optional /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1611398Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping __repr__ /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1611974Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping _type_check /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1612566Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping _type_convert /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1613144Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping __init__ /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1613694Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping __eq__ /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1614260Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping __hash__ /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1614830Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping Union /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1615413Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping <genexpr> /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1616003Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping _remove_dups_flatten /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1616603Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping _deduplicate /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1617234Z [2022-11-23 02:24:52,784] torch._dynamo.eval_frame: [DEBUG] skipping __init__ /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1617887Z [2022-11-23 02:24:52,785] torch._dynamo.eval_frame: [DEBUG] skipping __init__ /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1618499Z [2022-11-23 02:24:52,785] torch._dynamo.eval_frame: [DEBUG] skipping __setattr__ /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1619173Z [2022-11-23 02:24:52,785] torch._dynamo.eval_frame: [DEBUG] skipping _is_dunder /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1619925Z [2022-11-23 02:24:52,785] torch._dynamo.eval_frame: [DEBUG] skipping <genexpr> /opt/conda/lib/python3.10/typing.py
2022-11-23T02:25:07.1620799Z [2022-11-23 02:24:52,785] torch._dynamo.convert_frame: [DEBUG] skipping because no torch.* _collect_type_vars             /opt/conda/lib/python3.10/site-packages/typing_extensions.py 123
2022-11-23T02:25:07.1621718Z [2022-11-23 02:24:52,785] torch._dynamo.convert_frame: [DEBUG] skipping because no torch.* _should_collect_from_parameters             /opt/conda/lib/python3.10/site-packages/typing_extensions.py 111
2022-11-23T02:25:07.1622549Z [2022-11-23 02:24:52,785] torch._dynamo.eval_frame: [DEBUG] skipping annotate /opt/conda/lib/python3.10/site-packages/torch/jit/__init__.py
2022-11-23T02:25:07.1623313Z [2022-11-23 02:24:52,785] torch._dynamo.eval_frame: [DEBUG] skipping __enter__ /opt/conda/lib/python3.10/site-packages/torch/autograd/profiler.py
2022-11-23T02:25:07.1624001Z [2022-11-23 02:24:52,785] torch._dynamo.eval_frame: [DEBUG] skipping __call__ /opt/conda/lib/python3.10/site-packages/torch/_ops.py
2022-11-23T02:25:07.1624799Z [2022-11-23 02:24:52,785] torch._dynamo.eval_frame: [DEBUG] skipping __setattr__ /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py
2022-11-23T02:25:07.1625579Z [2022-11-23 02:24:52,785] torch._dynamo.eval_frame: [DEBUG] skipping __instancecheck__ /opt/conda/lib/python3.10/site-packages/torch/nn/parameter.py
2022-11-23T02:25:07.1645996Z [2022-11-23 02:24:52,786] torch._dynamo.eval_frame: [DEBUG] skipping notify_join_context /opt/conda/lib/python3.10/site-packages/torch/distributed/algorithms/join.py
2022-11-23T02:25:07.1647143Z [2022-11-23 02:24:52,786] torch._dynamo.eval_frame: [DEBUG] skipping __getattr__ /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py
2022-11-23T02:25:07.1647890Z [2022-11-23 02:24:52,786] torch._dynamo.eval_frame: [DEBUG] skipping _check_sync_bufs_pre_fwd /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py
2022-11-23T02:25:07.1648660Z [2022-11-23 02:24:52,786] torch._dynamo.eval_frame: [DEBUG] skipping will_sync_module_buffers /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py
2022-11-23T02:25:07.1649402Z [2022-11-23 02:24:52,786] torch._dynamo.eval_frame: [DEBUG] skipping _run_ddp_forward /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py
2022-11-23T02:25:07.1650125Z [2022-11-23 02:24:52,786] torch._dynamo.eval_frame: [DEBUG] skipping _inside_ddp_forward /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py
2022-11-23T02:25:07.1650764Z [2022-11-23 02:24:52,788] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
2022-11-23T02:25:07.1651435Z [2022-11-23 02:24:52,788] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:54
2022-11-23T02:25:07.1652019Z [2022-11-23 02:24:52,788] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
2022-11-23T02:25:07.1652549Z [2022-11-23 02:24:52,788] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR net [NNModuleVariable()]
2022-11-23T02:25:07.1653177Z [2022-11-23 02:24:52,788] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs [NNModuleVariable()]
2022-11-23T02:25:07.1653794Z [2022-11-23 02:24:52,789] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
2022-11-23T02:25:07.1654388Z [2022-11-23 02:24:52,827] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
2022-11-23T02:25:07.1654939Z [2022-11-23 02:24:52,827] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward
2022-11-23T02:25:07.1655504Z [2022-11-23 02:24:52,828] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
2022-11-23T02:25:07.1656164Z [2022-11-23 02:24:52,829] torch._dynamo.optimizations.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
2022-11-23T02:25:07.1656994Z [2022-11-23 02:24:52,829] torch._dynamo.optimizations.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
2022-11-23T02:25:07.1657591Z [2022-11-23 02:24:52,832] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.1657959Z ---orig graph---
2022-11-23T02:25:07.1658198Z graph():
2022-11-23T02:25:07.1658489Z     %inputs : torch.Tensor [#users=1] = placeholder[target=inputs]
2022-11-23T02:25:07.1658904Z     %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {})
2022-11-23T02:25:07.1659331Z     %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {})
2022-11-23T02:25:07.1659761Z     %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {})
2022-11-23T02:25:07.1660168Z     %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {})
2022-11-23T02:25:07.1660588Z     %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {})
2022-11-23T02:25:07.1660999Z     %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {})
2022-11-23T02:25:07.1661399Z     %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {})
2022-11-23T02:25:07.1661817Z     %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {})
2022-11-23T02:25:07.1662141Z     return (self_net_7,)
2022-11-23T02:25:07.1662307Z 
2022-11-23T02:25:07.1662462Z ---split graph---
2022-11-23T02:25:07.1662683Z graph():
2022-11-23T02:25:07.1662988Z     %inputs : torch.Tensor [#users=1] = placeholder[target=inputs]
2022-11-23T02:25:07.1663474Z     %submod_0 : [#users=1] = call_module[target=submod_0](args = (%inputs,), kwargs = {})
2022-11-23T02:25:07.1663871Z     %submod_1 : [#users=1] = call_module[target=submod_1](args = (%submod_0,), kwargs = {})
2022-11-23T02:25:07.1664275Z     %submod_2 : [#users=1] = call_module[target=submod_2](args = (%submod_1,), kwargs = {})
2022-11-23T02:25:07.1664597Z     return (submod_2,)
2022-11-23T02:25:07.1664759Z 
2022-11-23T02:25:07.1664899Z ---submod_0 graph---
2022-11-23T02:25:07.1665141Z graph():
2022-11-23T02:25:07.1665446Z     %inputs : torch.Tensor [#users=1] = placeholder[target=inputs]
2022-11-23T02:25:07.1665853Z     %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {})
2022-11-23T02:25:07.1666255Z     %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {})
2022-11-23T02:25:07.1666583Z     return self_net_1
2022-11-23T02:25:07.1666740Z 
2022-11-23T02:25:07.1667358Z ---submod_1 graph---
2022-11-23T02:25:07.1667602Z graph():
2022-11-23T02:25:07.1667888Z     %self_net_1 : [#users=1] = placeholder[target=self_net_1]
2022-11-23T02:25:07.1668284Z     %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {})
2022-11-23T02:25:07.1668692Z     %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {})
2022-11-23T02:25:07.1669018Z     return self_net_3
2022-11-23T02:25:07.1669175Z 
2022-11-23T02:25:07.1669331Z ---submod_2 graph---
2022-11-23T02:25:07.1669570Z graph():
2022-11-23T02:25:07.1669841Z     %self_net_3 : [#users=1] = placeholder[target=self_net_3]
2022-11-23T02:25:07.1670233Z     %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {})
2022-11-23T02:25:07.1670650Z     %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {})
2022-11-23T02:25:07.1671050Z     %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {})
2022-11-23T02:25:07.1671467Z     %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {})
2022-11-23T02:25:07.1671786Z     return self_net_7
2022-11-23T02:25:07.1671942Z 
2022-11-23T02:25:07.1672082Z ---------------
2022-11-23T02:25:07.1672217Z 
2022-11-23T02:25:07.1672728Z [2022-11-23 02:24:52,832] torch._dynamo.optimizations.distributed: [DEBUG] run_node placeholder, inputs got args tuple()
2022-11-23T02:25:07.1673451Z [2022-11-23 02:24:52,832] torch._dynamo.optimizations.distributed: [DEBUG] run_node call_module, submod_0 got args tuple(T[torch.Size([20, 10])])
2022-11-23T02:25:07.1674028Z [2022-11-23 02:24:52,832] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.1674380Z ---submod_0 graph---
2022-11-23T02:25:07.1674617Z graph():
2022-11-23T02:25:07.1674919Z     %inputs : torch.Tensor [#users=1] = placeholder[target=inputs]
2022-11-23T02:25:07.1675329Z     %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {})
2022-11-23T02:25:07.1675742Z     %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {})
2022-11-23T02:25:07.1676070Z     return self_net_1
2022-11-23T02:25:07.1676631Z [2022-11-23 02:24:52,889] torch._dynamo.optimizations.distributed: [DEBUG] run_node call_module, submod_1 got args tuple(T[torch.Size([20, 5000])])
2022-11-23T02:25:07.1677193Z [2022-11-23 02:24:52,889] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.1677570Z ---submod_1 graph---
2022-11-23T02:25:07.1677806Z graph():
2022-11-23T02:25:07.1678078Z     %self_net_1 : [#users=1] = placeholder[target=self_net_1]
2022-11-23T02:25:07.1678471Z     %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {})
2022-11-23T02:25:07.1678895Z     %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {})
2022-11-23T02:25:07.1679222Z     return self_net_3
2022-11-23T02:25:07.1679764Z [2022-11-23 02:24:52,941] torch._dynamo.optimizations.distributed: [DEBUG] run_node call_module, submod_2 got args tuple(T[torch.Size([20, 5000])])
2022-11-23T02:25:07.1680438Z [2022-11-23 02:24:52,941] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.1680809Z ---submod_2 graph---
2022-11-23T02:25:07.1681903Z graph():
2022-11-23T02:25:07.1682204Z     %self_net_3 : [#users=1] = placeholder[target=self_net_3]
2022-11-23T02:25:07.1682601Z     %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {})
2022-11-23T02:25:07.1683025Z     %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {})
2022-11-23T02:25:07.1683431Z     %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {})
2022-11-23T02:25:07.1683847Z     %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {})
2022-11-23T02:25:07.1684169Z     return self_net_7
2022-11-23T02:25:07.1684722Z [2022-11-23 02:24:53,034] torch._dynamo.optimizations.distributed: [DEBUG] run_node output, output got args tuple(tuple(T[torch.Size([20, 5])]))
2022-11-23T02:25:07.1685311Z [2022-11-23 02:24:53,034] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.1685675Z ---final graph---
2022-11-23T02:25:07.1685908Z graph():
2022-11-23T02:25:07.1686194Z     %inputs : torch.Tensor [#users=1] = placeholder[target=inputs]
2022-11-23T02:25:07.1686615Z     %submod_0 : [#users=1] = call_module[target=compiled_submod_0](args = (%inputs,), kwargs = {})
2022-11-23T02:25:07.1687063Z     %submod_1 : [#users=1] = call_module[target=compiled_submod_1](args = (%submod_0,), kwargs = {})
2022-11-23T02:25:07.1687488Z     %submod_2 : [#users=1] = call_module[target=compiled_submod_2](args = (%submod_1,), kwargs = {})
2022-11-23T02:25:07.1687829Z     return (submod_2,)
2022-11-23T02:25:07.1688108Z ---------------
2022-11-23T02:25:07.1688261Z 
2022-11-23T02:25:07.1688568Z [2022-11-23 02:24:53,034] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
2022-11-23T02:25:07.1689069Z [2022-11-23 02:24:53,035] torch._dynamo.output_graph: [CODE] TRACED GRAPH
2022-11-23T02:25:07.1689450Z  __compiled_fn_0 <eval_with_key>.1 opcode, name, target, args, kwargs
2022-11-23T02:25:07.1689795Z placeholder, inputs, inputs, (), {}
2022-11-23T02:25:07.1690099Z call_module, self_net_0, self_net_0, (inputs,), {}
2022-11-23T02:25:07.1690513Z call_module, self_net_1, self_net_1, (self_net_0,), {}
2022-11-23T02:25:07.1690861Z call_module, self_net_2, self_net_2, (self_net_1,), {}
2022-11-23T02:25:07.1691180Z call_module, self_net_3, self_net_3, (self_net_2,), {}
2022-11-23T02:25:07.1691512Z call_module, self_net_4, self_net_4, (self_net_3,), {}
2022-11-23T02:25:07.1691837Z call_module, self_net_5, self_net_5, (self_net_4,), {}
2022-11-23T02:25:07.1692145Z call_module, self_net_6, self_net_6, (self_net_5,), {}
2022-11-23T02:25:07.1692473Z call_module, self_net_7, self_net_7, (self_net_6,), {}
2022-11-23T02:25:07.1692789Z output, output, output, ((self_net_7,),), {}
2022-11-23T02:25:07.1692991Z 
2022-11-23T02:25:07.1693462Z [2022-11-23 02:24:53,035] torch._dynamo.convert_frame: [CODE] ORIGINAL BYTECODE forward /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 53 
2022-11-23T02:25:07.1693898Z  54           0 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1694187Z               2 LOAD_METHOD              0 (net)
2022-11-23T02:25:07.1694486Z               4 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.1694750Z               6 CALL_METHOD              1
2022-11-23T02:25:07.1695012Z               8 RETURN_VALUE
2022-11-23T02:25:07.1695174Z 
2022-11-23T02:25:07.1695265Z  
2022-11-23T02:25:07.1695836Z [2022-11-23 02:24:53,035] torch._dynamo.convert_frame: [CODE] MODIFIED BYTECODE forward /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 53 
2022-11-23T02:25:07.1696287Z  53           0 LOAD_GLOBAL              1 (__compiled_fn_0)
2022-11-23T02:25:07.1696590Z               2 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.1696946Z               4 CALL_FUNCTION            1
2022-11-23T02:25:07.1697212Z               6 UNPACK_SEQUENCE          1
2022-11-23T02:25:07.1697478Z               8 RETURN_VALUE
2022-11-23T02:25:07.1697635Z 
2022-11-23T02:25:07.1697726Z  
2022-11-23T02:25:07.1698077Z [2022-11-23 02:24:53,036] torch._dynamo.convert_frame: [CODE] GUARDS:
2022-11-23T02:25:07.1698376Z  - 
2022-11-23T02:25:07.1698660Z             local 'self' NN_MODULE
2022-11-23T02:25:07.1698889Z             {
2022-11-23T02:25:07.1699197Z                 'guard_types': ['ID_MATCH'],
2022-11-23T02:25:07.1699580Z                 'code': ['___check_obj_id(self, 140294324706352)'],
2022-11-23T02:25:07.1700026Z                 'obj_weakref': <weakref at 0x7f98ad728770; to 'ToyModel' at 0x7f98d1635c30>
2022-11-23T02:25:07.1700547Z                 'guarded_class': <weakref at 0x7f9a4b8d0630; to 'type' at 0x55a4264bb360 (ToyModel)>
2022-11-23T02:25:07.1700863Z             }
2022-11-23T02:25:07.1701080Z             
2022-11-23T02:25:07.1701294Z  - 
2022-11-23T02:25:07.1701583Z             local 'inputs' TENSOR_MATCH
2022-11-23T02:25:07.1701836Z             {
2022-11-23T02:25:07.1702137Z                 'guard_types': ['TENSOR_MATCH'],
2022-11-23T02:25:07.1702450Z                 'code': None,
2022-11-23T02:25:07.1702878Z                 'obj_weakref': <weakref at 0x7f98bc6eaca0; to 'Tensor' at 0x7f98da2a7d30>
2022-11-23T02:25:07.1703411Z                 'guarded_class': <weakref at 0x7f98dc435d50; to 'torch._C._TensorMeta' at 0x55a4248e47a0 (Tensor)>
2022-11-23T02:25:07.1703744Z             }
2022-11-23T02:25:07.1703960Z             
2022-11-23T02:25:07.1704172Z  - 
2022-11-23T02:25:07.1704485Z             local_nn_module 'self.net' NN_MODULE
2022-11-23T02:25:07.1704747Z             {
2022-11-23T02:25:07.1705020Z                 'guard_types': None,
2022-11-23T02:25:07.1705318Z                 'code': None,
2022-11-23T02:25:07.1705619Z                 'obj_weakref': None
2022-11-23T02:25:07.1705915Z                 'guarded_class': None
2022-11-23T02:25:07.1706165Z             }
2022-11-23T02:25:07.1706381Z             
2022-11-23T02:25:07.1706591Z  - 
2022-11-23T02:25:07.1707328Z             local_nn_module 'self.net[0]' NN_MODULE
2022-11-23T02:25:07.1707619Z             {
2022-11-23T02:25:07.1707904Z                 'guard_types': None,
2022-11-23T02:25:07.1708204Z                 'code': None,
2022-11-23T02:25:07.1708615Z                 'obj_weakref': None
2022-11-23T02:25:07.1708952Z                 'guarded_class': None
2022-11-23T02:25:07.1709181Z             }
2022-11-23T02:25:07.1709396Z             
2022-11-23T02:25:07.1709622Z  - 
2022-11-23T02:25:07.1709920Z             local_nn_module 'self.net[1]' NN_MODULE
2022-11-23T02:25:07.1710184Z             {
2022-11-23T02:25:07.1710470Z                 'guard_types': None,
2022-11-23T02:25:07.1710751Z                 'code': None,
2022-11-23T02:25:07.1711049Z                 'obj_weakref': None
2022-11-23T02:25:07.1711359Z                 'guarded_class': None
2022-11-23T02:25:07.1711589Z             }
2022-11-23T02:25:07.1711801Z             
2022-11-23T02:25:07.1712026Z  - 
2022-11-23T02:25:07.1712326Z             local_nn_module 'self.net[2]' NN_MODULE
2022-11-23T02:25:07.1712589Z             {
2022-11-23T02:25:07.1712877Z                 'guard_types': None,
2022-11-23T02:25:07.1713155Z                 'code': None,
2022-11-23T02:25:07.1713456Z                 'obj_weakref': None
2022-11-23T02:25:07.1713775Z                 'guarded_class': None
2022-11-23T02:25:07.1714004Z             }
2022-11-23T02:25:07.1714216Z             
2022-11-23T02:25:07.1714442Z  - 
2022-11-23T02:25:07.1714742Z             local_nn_module 'self.net[3]' NN_MODULE
2022-11-23T02:25:07.1715006Z             {
2022-11-23T02:25:07.1715292Z                 'guard_types': None,
2022-11-23T02:25:07.1715573Z                 'code': None,
2022-11-23T02:25:07.1715873Z                 'obj_weakref': None
2022-11-23T02:25:07.1716187Z                 'guarded_class': None
2022-11-23T02:25:07.1716413Z             }
2022-11-23T02:25:07.1716718Z             
2022-11-23T02:25:07.1716948Z  - 
2022-11-23T02:25:07.1717248Z             local_nn_module 'self.net[4]' NN_MODULE
2022-11-23T02:25:07.1717509Z             {
2022-11-23T02:25:07.1717796Z                 'guard_types': None,
2022-11-23T02:25:07.1718080Z                 'code': None,
2022-11-23T02:25:07.1718378Z                 'obj_weakref': None
2022-11-23T02:25:07.1718695Z                 'guarded_class': None
2022-11-23T02:25:07.1718922Z             }
2022-11-23T02:25:07.1719136Z             
2022-11-23T02:25:07.1719360Z  - 
2022-11-23T02:25:07.1719657Z             local_nn_module 'self.net[5]' NN_MODULE
2022-11-23T02:25:07.1719917Z             {
2022-11-23T02:25:07.1720202Z                 'guard_types': None,
2022-11-23T02:25:07.1720481Z                 'code': None,
2022-11-23T02:25:07.1720778Z                 'obj_weakref': None
2022-11-23T02:25:07.1721090Z                 'guarded_class': None
2022-11-23T02:25:07.1721332Z             }
2022-11-23T02:25:07.1721527Z             
2022-11-23T02:25:07.1721758Z  - 
2022-11-23T02:25:07.1722076Z             local_nn_module 'self.net[6]' NN_MODULE
2022-11-23T02:25:07.1722323Z             {
2022-11-23T02:25:07.1722611Z                 'guard_types': None,
2022-11-23T02:25:07.1722909Z                 'code': None,
2022-11-23T02:25:07.1723189Z                 'obj_weakref': None
2022-11-23T02:25:07.1723508Z                 'guarded_class': None
2022-11-23T02:25:07.1723751Z             }
2022-11-23T02:25:07.1723946Z             
2022-11-23T02:25:07.1724171Z  - 
2022-11-23T02:25:07.1724482Z             local_nn_module 'self.net[7]' NN_MODULE
2022-11-23T02:25:07.1724724Z             {
2022-11-23T02:25:07.1725009Z                 'guard_types': None,
2022-11-23T02:25:07.1725311Z                 'code': None,
2022-11-23T02:25:07.1725595Z                 'obj_weakref': None
2022-11-23T02:25:07.1725911Z                 'guarded_class': None
2022-11-23T02:25:07.1726153Z             }
2022-11-23T02:25:07.1726350Z             
2022-11-23T02:25:07.1726899Z [2022-11-23 02:24:53,036] torch._dynamo.eval_frame: [DEBUG] skipping forward /opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
2022-11-23T02:25:07.1727594Z [2022-11-23 02:24:53,036] torch._dynamo.eval_frame: [DEBUG] skipping __getattr__ /opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
2022-11-23T02:25:07.1728347Z [2022-11-23 02:24:53,036] torch._dynamo.eval_frame: [DEBUG] skipping __call__ /opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
2022-11-23T02:25:07.1729045Z [2022-11-23 02:24:53,036] torch._dynamo.eval_frame: [DEBUG] skipping innermost_fn /opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
2022-11-23T02:25:07.1729668Z [2022-11-23 02:24:53,036] torch._dynamo.eval_frame: [DEBUG] skipping wraps /opt/conda/lib/python3.10/functools.py
2022-11-23T02:25:07.1730275Z [2022-11-23 02:24:53,036] torch._dynamo.eval_frame: [DEBUG] skipping update_wrapper /opt/conda/lib/python3.10/functools.py
2022-11-23T02:25:07.1730929Z [2022-11-23 02:24:53,037] torch._dynamo.eval_frame: [DEBUG] skipping _fn /opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
2022-11-23T02:25:07.1731589Z [2022-11-23 02:24:53,037] torch._dynamo.eval_frame: [DEBUG] skipping nothing /opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
2022-11-23T02:25:07.1732215Z [2022-11-23 02:24:53,037] torch._dynamo.eval_frame: [DEBUG] skipping __exit__ /opt/conda/lib/python3.10/contextlib.py
2022-11-23T02:25:07.1732808Z [2022-11-23 02:24:53,037] torch._dynamo.eval_frame: [DEBUG] skipping __exit__ /opt/conda/lib/python3.10/contextlib.py
2022-11-23T02:25:07.1733494Z [2022-11-23 02:24:53,037] torch._dynamo.eval_frame: [DEBUG] skipping _check_sync_bufs_post_fwd /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py
2022-11-23T02:25:07.1734190Z [2022-11-23 02:24:53,037] torch._dynamo.eval_frame: [DEBUG] skipping __exit__ /opt/conda/lib/python3.10/site-packages/torch/autograd/profiler.py
2022-11-23T02:25:07.1734873Z [2022-11-23 02:24:53,038] torch._dynamo.eval_frame: [DEBUG] skipping is_scripting /opt/conda/lib/python3.10/site-packages/torch/_jit_internal.py
2022-11-23T02:25:07.1735626Z [2022-11-23 02:24:53,038] torch._dynamo.eval_frame: [DEBUG] skipping __getattr__ /opt/conda/lib/python3.10/site-packages/torch/_ops.py
2022-11-23T02:25:07.1736271Z [2022-11-23 02:24:53,038] torch._dynamo.eval_frame: [DEBUG] skipping __init__ /opt/conda/lib/python3.10/site-packages/torch/_ops.py
2022-11-23T02:25:07.1736882Z [2022-11-23 02:24:53,038] torch._dynamo.eval_frame: [DEBUG] skipping __call__ /opt/conda/lib/python3.10/site-packages/torch/_ops.py
2022-11-23T02:25:07.1737246Z ok (2.880s)
2022-11-23T02:25:07.1737528Z   test_custom_layer (__main__.TestDistributed)
2022-11-23T02:25:07.1738169Z Just ensures that the appropriate number of splits happen (based on ... [2022-11-23 02:24:53,069] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing opt_fn
2022-11-23T02:25:07.1738925Z [2022-11-23 02:24:53,069] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:455
2022-11-23T02:25:07.1739524Z [2022-11-23 02:24:53,069] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_DEREF ddp_m []
2022-11-23T02:25:07.1740181Z [2022-11-23 02:24:53,069] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs [UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.1740969Z [2022-11-23 02:24:53,069] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [UnspecializedNNModuleVariable(DistributedDataParallel), TensorVariable()]
2022-11-23T02:25:07.1741822Z [2022-11-23 02:24:53,071] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7f98dbcd87c0, file "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1056> 
2022-11-23T02:25:07.1742315Z  1057           0 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1742612Z                2 LOAD_ATTR                1 (autograd)
2022-11-23T02:25:07.1742891Z                4 LOAD_ATTR                2 (profiler)
2022-11-23T02:25:07.1743206Z                6 LOAD_METHOD              3 (record_function)
2022-11-23T02:25:07.1743400Z 
2022-11-23T02:25:07.1743686Z 1058           8 LOAD_CONST               1 ('DistributedDataParallel.forward')
2022-11-23T02:25:07.1743927Z 
2022-11-23T02:25:07.1744053Z 1057          10 CALL_METHOD              1
2022-11-23T02:25:07.1744381Z               12 SETUP_WITH             147 (to 308)
2022-11-23T02:25:07.1744659Z               14 POP_TOP
2022-11-23T02:25:07.1744815Z 
2022-11-23T02:25:07.1744949Z 1060          16 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1745238Z               18 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.1745528Z               20 CALL_METHOD              0
2022-11-23T02:25:07.1745814Z               22 POP_JUMP_IF_FALSE       39 (to 78)
2022-11-23T02:25:07.1746108Z               24 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1746409Z               26 LOAD_ATTR                5 (require_backward_grad_sync)
2022-11-23T02:25:07.1746738Z               28 POP_JUMP_IF_FALSE       39 (to 78)
2022-11-23T02:25:07.1747430Z 
2022-11-23T02:25:07.1747571Z 1061          30 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1747846Z               32 LOAD_ATTR                6 (logger)
2022-11-23T02:25:07.1748131Z               34 LOAD_CONST               0 (None)
2022-11-23T02:25:07.1748413Z               36 IS_OP                    1
2022-11-23T02:25:07.1748683Z               38 POP_JUMP_IF_TRUE        22 (to 44)
2022-11-23T02:25:07.1748967Z               40 LOAD_ASSERTION_ERROR
2022-11-23T02:25:07.1749246Z               42 RAISE_VARARGS            1
2022-11-23T02:25:07.1749419Z 
2022-11-23T02:25:07.1749549Z 1062     >>   44 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1749817Z               46 LOAD_ATTR                6 (logger)
2022-11-23T02:25:07.1750131Z               48 LOAD_METHOD              7 (set_runtime_stats_and_log)
2022-11-23T02:25:07.1750440Z               50 CALL_METHOD              0
2022-11-23T02:25:07.1750793Z               52 POP_TOP
2022-11-23T02:25:07.1750949Z 
2022-11-23T02:25:07.1751080Z 1063          54 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1751336Z               56 DUP_TOP
2022-11-23T02:25:07.1751601Z               58 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.1751897Z               60 LOAD_CONST               2 (1)
2022-11-23T02:25:07.1752166Z               62 INPLACE_ADD
2022-11-23T02:25:07.1752391Z               64 ROT_TWO
2022-11-23T02:25:07.1752663Z               66 STORE_ATTR               8 (num_iterations)
2022-11-23T02:25:07.1752893Z 
2022-11-23T02:25:07.1753026Z 1064          68 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1753314Z               70 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1753612Z               72 LOAD_METHOD             10 (prepare_for_forward)
2022-11-23T02:25:07.1753917Z               74 CALL_METHOD              0
2022-11-23T02:25:07.1754170Z               76 POP_TOP
2022-11-23T02:25:07.1754309Z 
2022-11-23T02:25:07.1754443Z 1068     >>   78 LOAD_GLOBAL             11 (Join)
2022-11-23T02:25:07.1754750Z               80 LOAD_METHOD             12 (notify_join_context)
2022-11-23T02:25:07.1755050Z               82 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1755310Z               84 CALL_METHOD              1
2022-11-23T02:25:07.1755592Z               86 STORE_FAST               3 (work)
2022-11-23T02:25:07.1755766Z 
2022-11-23T02:25:07.1755895Z 1069          88 LOAD_FAST                3 (work)
2022-11-23T02:25:07.1756187Z               90 POP_JUMP_IF_FALSE       54 (to 108)
2022-11-23T02:25:07.1756354Z 
2022-11-23T02:25:07.1756484Z 1070          92 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1756767Z               94 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1757092Z               96 LOAD_METHOD             13 (_set_forward_pass_work_handle)
2022-11-23T02:25:07.1757302Z 
2022-11-23T02:25:07.1757414Z 1071          98 LOAD_FAST                3 (work)
2022-11-23T02:25:07.1757698Z              100 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1758012Z              102 LOAD_ATTR               14 (_divide_by_initial_world_size)
2022-11-23T02:25:07.1758216Z 
2022-11-23T02:25:07.1758343Z 1070         104 CALL_METHOD              2
2022-11-23T02:25:07.1758578Z              106 POP_TOP
2022-11-23T02:25:07.1758732Z 
2022-11-23T02:25:07.1758944Z 1080     >>  108 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1759264Z              110 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.1759543Z              112 CALL_METHOD              0
2022-11-23T02:25:07.1759836Z              114 POP_JUMP_IF_FALSE       71 (to 142)
2022-11-23T02:25:07.1760128Z              116 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1760397Z              118 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1760704Z              120 LOAD_METHOD             15 (_rebuild_buckets)
2022-11-23T02:25:07.1760999Z              122 CALL_METHOD              0
2022-11-23T02:25:07.1761279Z              124 POP_JUMP_IF_FALSE       71 (to 142)
2022-11-23T02:25:07.1761461Z 
2022-11-23T02:25:07.1761595Z 1081         126 LOAD_GLOBAL              6 (logger)
2022-11-23T02:25:07.1761881Z              128 LOAD_METHOD             16 (info)
2022-11-23T02:25:07.1762054Z 
2022-11-23T02:25:07.1762388Z 1082         130 LOAD_CONST               3 ('Reducer buckets have been rebuilt in this iteration.')
2022-11-23T02:25:07.1762633Z 
2022-11-23T02:25:07.1762743Z 1081         132 CALL_METHOD              1
2022-11-23T02:25:07.1763004Z              134 POP_TOP
2022-11-23T02:25:07.1763157Z 
2022-11-23T02:25:07.1763290Z 1084         136 LOAD_CONST               4 (True)
2022-11-23T02:25:07.1763573Z              138 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1763862Z              140 STORE_ATTR              17 (_has_rebuilt_buckets)
2022-11-23T02:25:07.1764057Z 
2022-11-23T02:25:07.1764195Z 1088     >>  142 LOAD_GLOBAL             18 (hasattr)
2022-11-23T02:25:07.1764561Z              144 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1764919Z              146 LOAD_CONST               5 ('buffer_hook')
2022-11-23T02:25:07.1765214Z              148 CALL_FUNCTION            2
2022-11-23T02:25:07.1765528Z              150 STORE_FAST               4 (buffer_hook_registered)
2022-11-23T02:25:07.1765728Z 
2022-11-23T02:25:07.1765844Z 1089         152 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1766158Z              154 LOAD_METHOD             19 (_check_sync_bufs_pre_fwd)
2022-11-23T02:25:07.1766463Z              156 CALL_METHOD              0
2022-11-23T02:25:07.1766755Z              158 POP_JUMP_IF_FALSE       84 (to 168)
2022-11-23T02:25:07.1766921Z 
2022-11-23T02:25:07.1767049Z 1090         160 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1767347Z              162 LOAD_METHOD             20 (_sync_buffers)
2022-11-23T02:25:07.1767637Z              164 CALL_METHOD              0
2022-11-23T02:25:07.1767874Z              166 POP_TOP
2022-11-23T02:25:07.1768031Z 
2022-11-23T02:25:07.1768160Z 1092     >>  168 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1768455Z              170 LOAD_ATTR               21 (_join_config)
2022-11-23T02:25:07.1768731Z              172 LOAD_ATTR               22 (enable)
2022-11-23T02:25:07.1769025Z              174 POP_JUMP_IF_FALSE       94 (to 188)
2022-11-23T02:25:07.1769209Z 
2022-11-23T02:25:07.1769398Z 1094         176 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1769744Z              178 LOAD_ATTR               23 (_check_global_requires_backward_grad_sync)
2022-11-23T02:25:07.1769967Z 
2022-11-23T02:25:07.1770083Z 1095         180 LOAD_CONST               6 (False)
2022-11-23T02:25:07.1770259Z 
2022-11-23T02:25:07.1770493Z 1094         182 LOAD_CONST               7 (('is_joined_rank',))
2022-11-23T02:25:07.1770795Z              184 CALL_FUNCTION_KW         1
2022-11-23T02:25:07.1771039Z              186 POP_TOP
2022-11-23T02:25:07.1771194Z 
2022-11-23T02:25:07.1771324Z 1098     >>  188 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1771632Z              190 LOAD_ATTR               24 (_run_ddp_forward)
2022-11-23T02:25:07.1771929Z              192 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.1772197Z              194 BUILD_MAP                0
2022-11-23T02:25:07.1772478Z              196 LOAD_FAST                2 (kwargs)
2022-11-23T02:25:07.1772815Z              198 DICT_MERGE               1
2022-11-23T02:25:07.1773088Z              200 CALL_FUNCTION_EX         1
2022-11-23T02:25:07.1773374Z              202 STORE_FAST               5 (output)
2022-11-23T02:25:07.1773550Z 
2022-11-23T02:25:07.1773678Z 1102         204 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1773977Z              206 LOAD_METHOD             25 (_check_sync_bufs_post_fwd)
2022-11-23T02:25:07.1774284Z              208 CALL_METHOD              0
2022-11-23T02:25:07.1774572Z              210 POP_JUMP_IF_FALSE      110 (to 220)
2022-11-23T02:25:07.1774755Z 
2022-11-23T02:25:07.1774884Z 1103         212 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1775172Z              214 LOAD_METHOD             20 (_sync_buffers)
2022-11-23T02:25:07.1775458Z              216 CALL_METHOD              0
2022-11-23T02:25:07.1775709Z              218 POP_TOP
2022-11-23T02:25:07.1775862Z 
2022-11-23T02:25:07.1775978Z 1105     >>  220 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1776286Z              222 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.1776578Z              224 CALL_METHOD              0
2022-11-23T02:25:07.1776853Z              226 POP_JUMP_IF_FALSE      144 (to 288)
2022-11-23T02:25:07.1777147Z              228 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1777464Z              230 LOAD_ATTR                5 (require_backward_grad_sync)
2022-11-23T02:25:07.1777788Z              232 POP_JUMP_IF_FALSE      144 (to 288)
2022-11-23T02:25:07.1777953Z 
2022-11-23T02:25:07.1778084Z 1106         234 LOAD_CONST               4 (True)
2022-11-23T02:25:07.1778366Z              236 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1778755Z              238 STORE_ATTR              26 (require_forward_param_sync)
2022-11-23T02:25:07.1778961Z 
2022-11-23T02:25:07.1779075Z 1112         240 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1779387Z              242 LOAD_ATTR               27 (find_unused_parameters)
2022-11-23T02:25:07.1779709Z              244 POP_JUMP_IF_FALSE      137 (to 274)
2022-11-23T02:25:07.1780004Z              246 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1780280Z              248 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.1780583Z              250 POP_JUMP_IF_TRUE       137 (to 274)
2022-11-23T02:25:07.1780765Z 
2022-11-23T02:25:07.1780894Z 1114         252 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1781164Z              254 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1781479Z              256 LOAD_METHOD             29 (prepare_for_backward)
2022-11-23T02:25:07.1781680Z 
2022-11-23T02:25:07.1781811Z 1115         258 LOAD_GLOBAL             30 (list)
2022-11-23T02:25:07.1782094Z              260 LOAD_GLOBAL             31 (_find_tensors)
2022-11-23T02:25:07.1782392Z              262 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1782673Z              264 CALL_FUNCTION            1
2022-11-23T02:25:07.1782951Z              266 CALL_FUNCTION            1
2022-11-23T02:25:07.1783102Z 
2022-11-23T02:25:07.1783230Z 1114         268 CALL_METHOD              1
2022-11-23T02:25:07.1783479Z              270 POP_TOP
2022-11-23T02:25:07.1783749Z              272 JUMP_FORWARD            10 (to 294)
2022-11-23T02:25:07.1783927Z 
2022-11-23T02:25:07.1784039Z 1118     >>  274 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1784324Z              276 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1784637Z              278 LOAD_METHOD             29 (prepare_for_backward)
2022-11-23T02:25:07.1784927Z              280 BUILD_LIST               0
2022-11-23T02:25:07.1785200Z              282 CALL_METHOD              1
2022-11-23T02:25:07.1785456Z              284 POP_TOP
2022-11-23T02:25:07.1785709Z              286 JUMP_FORWARD             3 (to 294)
2022-11-23T02:25:07.1785887Z 
2022-11-23T02:25:07.1786018Z 1120     >>  288 LOAD_CONST               6 (False)
2022-11-23T02:25:07.1786303Z              290 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1786673Z              292 STORE_ATTR              26 (require_forward_param_sync)
2022-11-23T02:25:07.1787213Z          >>  294 POP_BLOCK
2022-11-23T02:25:07.1787374Z 
2022-11-23T02:25:07.1787517Z 1057         296 LOAD_CONST               0 (None)
2022-11-23T02:25:07.1787776Z              298 DUP_TOP
2022-11-23T02:25:07.1787985Z              300 DUP_TOP
2022-11-23T02:25:07.1788246Z              302 CALL_FUNCTION            3
2022-11-23T02:25:07.1788503Z              304 POP_TOP
2022-11-23T02:25:07.1788755Z              306 JUMP_FORWARD             8 (to 324)
2022-11-23T02:25:07.1789032Z          >>  308 WITH_EXCEPT_START
2022-11-23T02:25:07.1789315Z              310 POP_JUMP_IF_TRUE       157 (to 314)
2022-11-23T02:25:07.1789588Z              312 RERAISE                  1
2022-11-23T02:25:07.1789836Z          >>  314 POP_TOP
2022-11-23T02:25:07.1790069Z              316 POP_TOP
2022-11-23T02:25:07.1790302Z              318 POP_TOP
2022-11-23T02:25:07.1790524Z              320 POP_EXCEPT
2022-11-23T02:25:07.1790766Z              322 POP_TOP
2022-11-23T02:25:07.1790917Z 
2022-11-23T02:25:07.1791050Z 1124     >>  324 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1791349Z              326 LOAD_ATTR               27 (find_unused_parameters)
2022-11-23T02:25:07.1791672Z              328 POP_JUMP_IF_FALSE      168 (to 336)
2022-11-23T02:25:07.1791960Z              330 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1792239Z              332 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.1792541Z              334 POP_JUMP_IF_FALSE      178 (to 356)
2022-11-23T02:25:07.1792727Z 
2022-11-23T02:25:07.1792858Z 1125     >>  336 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1793233Z              338 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.1793415Z 
2022-11-23T02:25:07.1793542Z 1124         340 EXTENDED_ARG             1
2022-11-23T02:25:07.1793831Z              342 POP_JUMP_IF_FALSE      265 (to 530)
2022-11-23T02:25:07.1794011Z 
2022-11-23T02:25:07.1794136Z 1125         344 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1794422Z              346 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.1794713Z              348 LOAD_CONST               2 (1)
2022-11-23T02:25:07.1794996Z              350 COMPARE_OP               2 (==)
2022-11-23T02:25:07.1795171Z 
2022-11-23T02:25:07.1795281Z 1124         352 EXTENDED_ARG             1
2022-11-23T02:25:07.1795574Z              354 POP_JUMP_IF_FALSE      265 (to 530)
2022-11-23T02:25:07.1795756Z 
2022-11-23T02:25:07.1795882Z 1128     >>  356 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1796176Z              358 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.1796347Z 
2022-11-23T02:25:07.1796477Z 1129         360 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1796775Z              362 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.1796960Z 
2022-11-23T02:25:07.1797239Z 1127         364 LOAD_CONST               8 (('static_graph', 'num_iterations'))
2022-11-23T02:25:07.1797559Z              366 BUILD_CONST_KEY_MAP      2
2022-11-23T02:25:07.1797860Z              368 STORE_FAST               6 (state_dict)
2022-11-23T02:25:07.1798042Z 
2022-11-23T02:25:07.1798199Z 1136         370 LOAD_GLOBAL             32 (_tree_flatten_with_rref)
2022-11-23T02:25:07.1798508Z              372 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1798776Z              374 CALL_FUNCTION            1
2022-11-23T02:25:07.1798943Z 
2022-11-23T02:25:07.1799076Z 1132         376 UNPACK_SEQUENCE          3
2022-11-23T02:25:07.1799251Z 
2022-11-23T02:25:07.1799404Z 1133         378 STORE_FAST               7 (output_tensor_list)
2022-11-23T02:25:07.1799606Z 
2022-11-23T02:25:07.1799729Z 1134         380 STORE_FAST               8 (treespec)
2022-11-23T02:25:07.1799909Z 
2022-11-23T02:25:07.1800057Z 1135         382 STORE_FAST               9 (output_is_rref)
2022-11-23T02:25:07.1800242Z 
2022-11-23T02:25:07.1800805Z 1137         384 LOAD_CONST               9 (<code object <listcomp> at 0x7f98dbcd8660, file "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1137>)
2022-11-23T02:25:07.1801465Z              386 LOAD_CONST              10 ('DistributedDataParallel.forward.<locals>.<listcomp>')
2022-11-23T02:25:07.1801825Z              388 MAKE_FUNCTION            0
2022-11-23T02:25:07.1802111Z              390 LOAD_GLOBAL             33 (range)
2022-11-23T02:25:07.1802399Z              392 LOAD_GLOBAL             34 (len)
2022-11-23T02:25:07.1802707Z              394 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.1802991Z              396 CALL_FUNCTION            1
2022-11-23T02:25:07.1803275Z              398 CALL_FUNCTION            1
2022-11-23T02:25:07.1803527Z              400 GET_ITER
2022-11-23T02:25:07.1803771Z              402 CALL_FUNCTION            1
2022-11-23T02:25:07.1804077Z              404 STORE_FAST              10 (output_placeholders)
2022-11-23T02:25:07.1804277Z 
2022-11-23T02:25:07.1804417Z 1140         406 LOAD_GLOBAL             35 (enumerate)
2022-11-23T02:25:07.1804720Z              408 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.1805021Z              410 CALL_FUNCTION            1
2022-11-23T02:25:07.1805275Z              412 GET_ITER
2022-11-23T02:25:07.1805520Z          >>  414 FOR_ITER                18 (to 452)
2022-11-23T02:25:07.1805805Z              416 UNPACK_SEQUENCE          2
2022-11-23T02:25:07.1806087Z              418 STORE_FAST              11 (i)
2022-11-23T02:25:07.1806374Z              420 STORE_FAST               5 (output)
2022-11-23T02:25:07.1806534Z 
2022-11-23T02:25:07.1806669Z 1141         422 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1807038Z              424 LOAD_METHOD             36 (is_tensor)
2022-11-23T02:25:07.1807332Z              426 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1807599Z              428 CALL_METHOD              1
2022-11-23T02:25:07.1807895Z              430 POP_JUMP_IF_FALSE      225 (to 450)
2022-11-23T02:25:07.1808189Z              432 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1808466Z              434 LOAD_ATTR               37 (grad_fn)
2022-11-23T02:25:07.1808753Z              436 LOAD_CONST               0 (None)
2022-11-23T02:25:07.1809029Z              438 IS_OP                    0
2022-11-23T02:25:07.1809315Z              440 POP_JUMP_IF_FALSE      225 (to 450)
2022-11-23T02:25:07.1809480Z 
2022-11-23T02:25:07.1809611Z 1142         442 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1809921Z              444 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1810224Z              446 LOAD_FAST               11 (i)
2022-11-23T02:25:07.1810477Z              448 STORE_SUBSCR
2022-11-23T02:25:07.1810754Z          >>  450 JUMP_ABSOLUTE          207 (to 414)
2022-11-23T02:25:07.1810932Z 
2022-11-23T02:25:07.1811070Z 1149     >>  452 LOAD_GLOBAL             38 (_DDPSink)
2022-11-23T02:25:07.1811347Z              454 LOAD_ATTR               39 (apply)
2022-11-23T02:25:07.1811523Z 
2022-11-23T02:25:07.1811655Z 1150         456 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1811943Z              458 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1812119Z 
2022-11-23T02:25:07.1812258Z 1151         460 LOAD_FAST                6 (state_dict)
2022-11-23T02:25:07.1812435Z 
2022-11-23T02:25:07.1812544Z 1149         462 BUILD_LIST               2
2022-11-23T02:25:07.1812713Z 
2022-11-23T02:25:07.1812865Z 1152         464 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.1813054Z 
2022-11-23T02:25:07.1813181Z 1149         466 LIST_EXTEND              1
2022-11-23T02:25:07.1813443Z              468 LIST_TO_TUPLE
2022-11-23T02:25:07.1813701Z              470 CALL_FUNCTION_EX         0
2022-11-23T02:25:07.1814021Z              472 STORE_FAST              12 (passthrough_tensor_list)
2022-11-23T02:25:07.1814226Z 
2022-11-23T02:25:07.1814360Z 1154         474 LOAD_GLOBAL             33 (range)
2022-11-23T02:25:07.1814631Z              476 LOAD_GLOBAL             34 (len)
2022-11-23T02:25:07.1815004Z              478 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1815322Z              480 CALL_FUNCTION            1
2022-11-23T02:25:07.1815581Z              482 CALL_FUNCTION            1
2022-11-23T02:25:07.1815841Z              484 GET_ITER
2022-11-23T02:25:07.1816105Z          >>  486 FOR_ITER                15 (to 518)
2022-11-23T02:25:07.1816372Z              488 STORE_FAST              11 (i)
2022-11-23T02:25:07.1816544Z 
2022-11-23T02:25:07.1816702Z 1155         490 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1817008Z              492 LOAD_FAST               11 (i)
2022-11-23T02:25:07.1817273Z              494 BINARY_SUBSCR
2022-11-23T02:25:07.1817529Z              496 LOAD_CONST               0 (None)
2022-11-23T02:25:07.1817807Z              498 IS_OP                    0
2022-11-23T02:25:07.1818080Z              500 EXTENDED_ARG             1
2022-11-23T02:25:07.1818356Z              502 POP_JUMP_IF_FALSE      258 (to 516)
2022-11-23T02:25:07.1818540Z 
2022-11-23T02:25:07.1818710Z 1156         504 LOAD_FAST               12 (passthrough_tensor_list)
2022-11-23T02:25:07.1819024Z              506 LOAD_FAST               11 (i)
2022-11-23T02:25:07.1819272Z              508 BINARY_SUBSCR
2022-11-23T02:25:07.1819565Z              510 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1819865Z              512 LOAD_FAST               11 (i)
2022-11-23T02:25:07.1820124Z              514 STORE_SUBSCR
2022-11-23T02:25:07.1820385Z          >>  516 JUMP_ABSOLUTE          243 (to 486)
2022-11-23T02:25:07.1820564Z 
2022-11-23T02:25:07.1820728Z 1159     >>  518 LOAD_GLOBAL             40 (_tree_unflatten_with_rref)
2022-11-23T02:25:07.1820995Z 
2022-11-23T02:25:07.1821153Z 1160         520 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1821450Z              522 LOAD_FAST                8 (treespec)
2022-11-23T02:25:07.1821756Z              524 LOAD_FAST                9 (output_is_rref)
2022-11-23T02:25:07.1821942Z 
2022-11-23T02:25:07.1822074Z 1159         526 CALL_FUNCTION            3
2022-11-23T02:25:07.1822362Z              528 STORE_FAST               5 (output)
2022-11-23T02:25:07.1822522Z 
2022-11-23T02:25:07.1822654Z 1162     >>  530 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1822917Z              532 RETURN_VALUE
2022-11-23T02:25:07.1823146Z  
2022-11-23T02:25:07.1823256Z 
2022-11-23T02:25:07.1823728Z [2022-11-23 02:24:53,073] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1057
2022-11-23T02:25:07.1824339Z [2022-11-23 02:24:53,073] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
2022-11-23T02:25:07.1825058Z [2022-11-23 02:24:53,073] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR autograd [TorchVariable(<module 'torch' from '/opt/conda/lib/python3.10/site-packages/torch/__init__.py'>)]
2022-11-23T02:25:07.1825927Z [2022-11-23 02:24:53,073] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR profiler [TorchVariable(<module 'torch.autograd' from '/opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py'>)]
2022-11-23T02:25:07.1827015Z [2022-11-23 02:24:53,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR record_function [TorchVariable(<module 'torch.autograd.profiler' from '/opt/conda/lib/python3.10/site-packages/torch/autograd/profiler.py'>)]
2022-11-23T02:25:07.1827874Z [2022-11-23 02:24:53,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1058
2022-11-23T02:25:07.1828697Z [2022-11-23 02:24:53,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST DistributedDataParallel.forward [TorchVariable(<class 'torch.autograd.profiler.record_function'>)]
2022-11-23T02:25:07.1829510Z [2022-11-23 02:24:53,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1057
2022-11-23T02:25:07.1830386Z [2022-11-23 02:24:53,074] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TorchVariable(<class 'torch.autograd.profiler.record_function'>), ConstantVariable(str)]
2022-11-23T02:25:07.1831026Z [2022-11-23 02:24:53,074] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored
2022-11-23T02:25:07.1831595Z [2022-11-23 02:24:53,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE SETUP_WITH 308 [NullContextVariable()]
2022-11-23T02:25:07.1832254Z [2022-11-23 02:24:53,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType)]
2022-11-23T02:25:07.1832987Z [2022-11-23 02:24:53,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1060
2022-11-23T02:25:07.1833637Z [2022-11-23 02:24:53,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [WithExitFunctionVariable()]
2022-11-23T02:25:07.1834465Z [2022-11-23 02:24:53,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR is_grad_enabled [WithExitFunctionVariable(), TorchVariable(<module 'torch' from '/opt/conda/lib/python3.10/site-packages/torch/__init__.py'>)]
2022-11-23T02:25:07.1835306Z [2022-11-23 02:24:53,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), TorchVariable(<built-in function is_grad_enabled>)]
2022-11-23T02:25:07.1836032Z [2022-11-23 02:24:53,075] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 78 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.1836666Z [2022-11-23 02:24:53,076] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.1837566Z [2022-11-23 02:24:53,076] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR require_backward_grad_sync [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.1838375Z [2022-11-23 02:24:53,076] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 78 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.1839107Z [2022-11-23 02:24:53,076] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1061
2022-11-23T02:25:07.1839852Z [2022-11-23 02:24:53,076] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.1840590Z [2022-11-23 02:24:53,076] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR logger [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.1841378Z [2022-11-23 02:24:53,077] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger)]
2022-11-23T02:25:07.1842169Z [2022-11-23 02:24:53,077] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger), ConstantVariable(NoneType)]
2022-11-23T02:25:07.1842920Z [2022-11-23 02:24:53,077] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 44 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.1843639Z [2022-11-23 02:24:53,077] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1062
2022-11-23T02:25:07.1844293Z [2022-11-23 02:24:53,077] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.1845050Z [2022-11-23 02:24:53,077] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR logger [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.1845886Z [2022-11-23 02:24:53,078] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR set_runtime_stats_and_log [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger)]
2022-11-23T02:25:07.1846649Z [2022-11-23 02:24:53,079] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), UserDefinedObjectVariable(instancemethod)]
2022-11-23T02:25:07.1847479Z [2022-11-23 02:24:53,079] torch._dynamo.convert_frame: [CODE] ORIGINAL BYTECODE opt_fn /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 453 
2022-11-23T02:25:07.1847936Z 455           0 LOAD_DEREF               0 (ddp_m)
2022-11-23T02:25:07.1848230Z               2 LOAD_FAST                0 (inputs)
2022-11-23T02:25:07.1848496Z               4 CALL_FUNCTION            1
2022-11-23T02:25:07.1848760Z               6 RETURN_VALUE
2022-11-23T02:25:07.1848921Z 
2022-11-23T02:25:07.1849012Z  
2022-11-23T02:25:07.1849565Z [2022-11-23 02:24:53,080] torch._dynamo.convert_frame: [CODE] MODIFIED BYTECODE opt_fn /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 453 
2022-11-23T02:25:07.1850019Z 453           0 LOAD_DEREF               0 (ddp_m)
2022-11-23T02:25:07.1850312Z               2 LOAD_FAST                0 (inputs)
2022-11-23T02:25:07.1850491Z 
2022-11-23T02:25:07.1850619Z 455           4 CALL_FUNCTION            1
2022-11-23T02:25:07.1850868Z               6 RETURN_VALUE
2022-11-23T02:25:07.1851030Z 
2022-11-23T02:25:07.1851121Z  
2022-11-23T02:25:07.1851485Z [2022-11-23 02:24:53,080] torch._dynamo.convert_frame: [CODE] GUARDS:
2022-11-23T02:25:07.1851766Z  - 
2022-11-23T02:25:07.1852052Z             local 'ddp_m' TYPE_MATCH
2022-11-23T02:25:07.1852302Z             {
2022-11-23T02:25:07.1852599Z                 'guard_types': ['TYPE_MATCH'],
2022-11-23T02:25:07.1853037Z                 'code': ['___check_type_id(ddp_m, 94163481642224)'],
2022-11-23T02:25:07.1853562Z                 'obj_weakref': <weakref at 0x7f98ad892ca0; to 'DistributedDataParallel' at 0x7f98bc70c4f0>
2022-11-23T02:25:07.1854250Z                 'guarded_class': <weakref at 0x7f98dbce1a80; to 'ABCMeta' at 0x55a424dfc4f0 (DistributedDataParallel)>
2022-11-23T02:25:07.1854587Z             }
2022-11-23T02:25:07.1854804Z             
2022-11-23T02:25:07.1855031Z  - 
2022-11-23T02:25:07.1855301Z             local 'inputs' TENSOR_MATCH
2022-11-23T02:25:07.1855557Z             {
2022-11-23T02:25:07.1855876Z                 'guard_types': ['TENSOR_MATCH'],
2022-11-23T02:25:07.1856177Z                 'code': None,
2022-11-23T02:25:07.1856608Z                 'obj_weakref': <weakref at 0x7f98ad8b6160; to 'Tensor' at 0x7f98ad892f70>
2022-11-23T02:25:07.1857152Z                 'guarded_class': <weakref at 0x7f98dc435d50; to 'torch._C._TensorMeta' at 0x55a4248e47a0 (Tensor)>
2022-11-23T02:25:07.1857465Z             }
2022-11-23T02:25:07.1857677Z             
2022-11-23T02:25:07.1858130Z [2022-11-23 02:24:53,082] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
2022-11-23T02:25:07.1858793Z [2022-11-23 02:24:53,082] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:92
2022-11-23T02:25:07.1859381Z [2022-11-23 02:24:53,082] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
2022-11-23T02:25:07.1859929Z [2022-11-23 02:24:53,082] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR seq [NNModuleVariable()]
2022-11-23T02:25:07.1860496Z [2022-11-23 02:24:53,083] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable()]
2022-11-23T02:25:07.1861080Z [2022-11-23 02:24:53,083] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
2022-11-23T02:25:07.1861859Z [2022-11-23 02:24:53,083] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7f9a4b8cb520, file "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 77> 
2022-11-23T02:25:07.1862341Z   78           0 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1862638Z               2 LOAD_METHOD              0 (linear)
2022-11-23T02:25:07.1862907Z               4 LOAD_FAST                1 (x)
2022-11-23T02:25:07.1863182Z               6 CALL_METHOD              1
2022-11-23T02:25:07.1863443Z               8 RETURN_VALUE
2022-11-23T02:25:07.1863656Z  
2022-11-23T02:25:07.1863785Z 
2022-11-23T02:25:07.1864279Z [2022-11-23 02:24:53,083] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:78
2022-11-23T02:25:07.1864881Z [2022-11-23 02:24:53,083] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
2022-11-23T02:25:07.1865428Z [2022-11-23 02:24:53,084] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR linear [NNModuleVariable()]
2022-11-23T02:25:07.1865977Z [2022-11-23 02:24:53,084] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable()]
2022-11-23T02:25:07.1866578Z [2022-11-23 02:24:53,084] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
2022-11-23T02:25:07.1867445Z [2022-11-23 02:24:53,091] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
2022-11-23T02:25:07.1868224Z [2022-11-23 02:24:53,091] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7f9a4b8cb520, file "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 77>
2022-11-23T02:25:07.1869068Z [2022-11-23 02:24:53,093] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7f9a4b8cb260, file "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 69> 
2022-11-23T02:25:07.1869546Z   70           0 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1869837Z               2 LOAD_METHOD              1 (mm)
2022-11-23T02:25:07.1870127Z               4 LOAD_FAST                1 (x)
2022-11-23T02:25:07.1870393Z               6 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1870792Z               8 LOAD_ATTR                2 (weight)
2022-11-23T02:25:07.1871078Z              10 LOAD_METHOD              3 (t)
2022-11-23T02:25:07.1871342Z              12 CALL_METHOD              0
2022-11-23T02:25:07.1871616Z              14 CALL_METHOD              2
2022-11-23T02:25:07.1871874Z              16 RETURN_VALUE
2022-11-23T02:25:07.1872089Z  
2022-11-23T02:25:07.1872219Z 
2022-11-23T02:25:07.1872668Z [2022-11-23 02:24:53,093] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:70
2022-11-23T02:25:07.1873265Z [2022-11-23 02:24:53,094] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
2022-11-23T02:25:07.1873965Z [2022-11-23 02:24:53,094] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mm [TorchVariable(<module 'torch' from '/opt/conda/lib/python3.10/site-packages/torch/__init__.py'>)]
2022-11-23T02:25:07.1874691Z [2022-11-23 02:24:53,094] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method mm of type object at 0x7f9905f08c60>)]
2022-11-23T02:25:07.1875438Z [2022-11-23 02:24:53,094] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TorchVariable(<built-in method mm of type object at 0x7f9905f08c60>), TensorVariable()]
2022-11-23T02:25:07.1876247Z [2022-11-23 02:24:53,094] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR weight [TorchVariable(<built-in method mm of type object at 0x7f9905f08c60>), TensorVariable(), NNModuleVariable()]
2022-11-23T02:25:07.1877070Z [2022-11-23 02:24:53,096] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR t [TorchVariable(<built-in method mm of type object at 0x7f9905f08c60>), TensorVariable(), TensorVariable()]
2022-11-23T02:25:07.1877937Z [2022-11-23 02:24:53,097] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method mm of type object at 0x7f9905f08c60>), TensorVariable(), GetAttrVariable(TensorVariable(), t)]
2022-11-23T02:25:07.1878774Z [2022-11-23 02:24:53,098] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TorchVariable(<built-in method mm of type object at 0x7f9905f08c60>), TensorVariable(), TensorVariable()]
2022-11-23T02:25:07.1879440Z [2022-11-23 02:24:53,099] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
2022-11-23T02:25:07.1880280Z [2022-11-23 02:24:53,099] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7f9a4b8cb260, file "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 69>
2022-11-23T02:25:07.1881153Z [2022-11-23 02:24:53,101] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7f9a4b8cb520, file "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 77> 
2022-11-23T02:25:07.1881610Z   78           0 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1881902Z               2 LOAD_METHOD              0 (linear)
2022-11-23T02:25:07.1882187Z               4 LOAD_FAST                1 (x)
2022-11-23T02:25:07.1882471Z               6 CALL_METHOD              1
2022-11-23T02:25:07.1882717Z               8 RETURN_VALUE
2022-11-23T02:25:07.1882951Z  
2022-11-23T02:25:07.1883081Z 
2022-11-23T02:25:07.1883517Z [2022-11-23 02:24:53,101] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:78
2022-11-23T02:25:07.1884091Z [2022-11-23 02:24:53,101] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
2022-11-23T02:25:07.1884642Z [2022-11-23 02:24:53,102] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR linear [NNModuleVariable()]
2022-11-23T02:25:07.1885210Z [2022-11-23 02:24:53,102] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable()]
2022-11-23T02:25:07.1885815Z [2022-11-23 02:24:53,102] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
2022-11-23T02:25:07.1886386Z [2022-11-23 02:24:53,109] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
2022-11-23T02:25:07.1887224Z [2022-11-23 02:24:53,109] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7f9a4b8cb520, file "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 77>
2022-11-23T02:25:07.1887899Z [2022-11-23 02:24:53,111] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
2022-11-23T02:25:07.1888467Z [2022-11-23 02:24:53,111] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward
2022-11-23T02:25:07.1889006Z [2022-11-23 02:24:53,113] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
2022-11-23T02:25:07.1889664Z [2022-11-23 02:24:53,113] torch._dynamo.optimizations.distributed: [INFO] DDPOptimizer used bucket cap 1048576 and produced the following buckets:
2022-11-23T02:25:07.1890392Z [2022-11-23 02:24:53,113] torch._dynamo.optimizations.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
2022-11-23T02:25:07.1890989Z [2022-11-23 02:24:53,116] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.1891340Z ---orig graph---
2022-11-23T02:25:07.1891575Z graph():
2022-11-23T02:25:07.1891863Z     %x : torch.Tensor [#users=1] = placeholder[target=x]
2022-11-23T02:25:07.1892260Z     %self_seq_0_linear : [#users=1] = call_module[target=self_seq_0_linear](args = (%x,), kwargs = {})
2022-11-23T02:25:07.1892706Z     %self_seq_1 : [#users=1] = call_module[target=self_seq_1](args = (%self_seq_0_linear,), kwargs = {})
2022-11-23T02:25:07.1893118Z     %self_seq_2_weight : [#users=1] = get_attr[target=self_seq_2_weight]
2022-11-23T02:25:07.1893496Z     %t : [#users=1] = call_method[target=t](args = (%self_seq_2_weight,), kwargs = {})
2022-11-23T02:25:07.1893906Z     %mm : [#users=1] = call_function[target=torch.mm](args = (%self_seq_1, %t), kwargs = {})
2022-11-23T02:25:07.1894318Z     %self_seq_3 : [#users=1] = call_module[target=self_seq_3](args = (%mm,), kwargs = {})
2022-11-23T02:25:07.1894556Z     %self_seq_4_linear : [#users=1] = call_module[target=self_seq_4_linear](args = (%self_seq_3,), kwargs = {})
2022-11-23T02:25:07.1894774Z     %self_seq_5 : [#users=1] = call_module[target=self_seq_5](args = (%self_seq_4_linear,), kwargs = {})
2022-11-23T02:25:07.1894890Z     return (self_seq_5,)
2022-11-23T02:25:07.1894910Z 
2022-11-23T02:25:07.1895042Z ---split graph---
2022-11-23T02:25:07.1895199Z graph():
2022-11-23T02:25:07.1895375Z     %x : torch.Tensor [#users=1] = placeholder[target=x]
2022-11-23T02:25:07.1895557Z     %self_seq_2_weight : [#users=1] = get_attr[target=self_seq_2_weight]
2022-11-23T02:25:07.1895753Z     %submod_0 : [#users=1] = call_module[target=submod_0](args = (%x,), kwargs = {})
2022-11-23T02:25:07.1895981Z     %submod_1 : [#users=1] = call_module[target=submod_1](args = (%self_seq_2_weight, %submod_0), kwargs = {})
2022-11-23T02:25:07.1896184Z     %submod_2 : [#users=1] = call_module[target=submod_2](args = (%submod_1,), kwargs = {})
2022-11-23T02:25:07.1896298Z     return (submod_2,)
2022-11-23T02:25:07.1896318Z 
2022-11-23T02:25:07.1896456Z ---submod_0 graph---
2022-11-23T02:25:07.1896552Z graph():
2022-11-23T02:25:07.1896717Z     %x : torch.Tensor [#users=1] = placeholder[target=x]
2022-11-23T02:25:07.1896935Z     %self_seq_0_linear : [#users=1] = call_module[target=self_seq_0_linear](args = (%x,), kwargs = {})
2022-11-23T02:25:07.1897159Z     %self_seq_1 : [#users=1] = call_module[target=self_seq_1](args = (%self_seq_0_linear,), kwargs = {})
2022-11-23T02:25:07.1897270Z     return self_seq_1
2022-11-23T02:25:07.1897289Z 
2022-11-23T02:25:07.1897438Z ---submod_1 graph---
2022-11-23T02:25:07.1897516Z graph():
2022-11-23T02:25:07.1897705Z     %self_seq_2_weight : [#users=1] = placeholder[target=self_seq_2_weight]
2022-11-23T02:25:07.1897874Z     %self_seq_1 : [#users=1] = placeholder[target=self_seq_1]
2022-11-23T02:25:07.1898069Z     %t : [#users=1] = call_method[target=t](args = (%self_seq_2_weight,), kwargs = {})
2022-11-23T02:25:07.1898274Z     %mm : [#users=1] = call_function[target=torch.mm](args = (%self_seq_1, %t), kwargs = {})
2022-11-23T02:25:07.1898529Z     %self_seq_3 : [#users=1] = call_module[target=self_seq_3](args = (%mm,), kwargs = {})
2022-11-23T02:25:07.1898644Z     return self_seq_3
2022-11-23T02:25:07.1898663Z 
2022-11-23T02:25:07.1898814Z ---submod_2 graph---
2022-11-23T02:25:07.1898892Z graph():
2022-11-23T02:25:07.1899065Z     %self_seq_3 : [#users=1] = placeholder[target=self_seq_3]
2022-11-23T02:25:07.1899294Z     %self_seq_4_linear : [#users=1] = call_module[target=self_seq_4_linear](args = (%self_seq_3,), kwargs = {})
2022-11-23T02:25:07.1899513Z     %self_seq_5 : [#users=1] = call_module[target=self_seq_5](args = (%self_seq_4_linear,), kwargs = {})
2022-11-23T02:25:07.1899623Z     return self_seq_5
2022-11-23T02:25:07.1899642Z 
2022-11-23T02:25:07.1899778Z ---------------
2022-11-23T02:25:07.1899797Z 
2022-11-23T02:25:07.1900166Z [2022-11-23 02:24:53,116] torch._dynamo.optimizations.distributed: [DEBUG] run_node placeholder, x got args tuple()
2022-11-23T02:25:07.1900555Z [2022-11-23 02:24:53,116] torch._dynamo.optimizations.distributed: [DEBUG] run_node get_attr, self_seq_2_weight got args tuple()
2022-11-23T02:25:07.1900976Z [2022-11-23 02:24:53,116] torch._dynamo.optimizations.distributed: [DEBUG] run_node call_module, submod_0 got args tuple(T[torch.Size([512, 512])])
2022-11-23T02:25:07.1901241Z [2022-11-23 02:24:53,116] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.1901388Z ---submod_0 graph---
2022-11-23T02:25:07.1901486Z graph():
2022-11-23T02:25:07.1901652Z     %x : torch.Tensor [#users=1] = placeholder[target=x]
2022-11-23T02:25:07.1901869Z     %self_seq_0_linear : [#users=1] = call_module[target=self_seq_0_linear](args = (%x,), kwargs = {})
2022-11-23T02:25:07.1902089Z     %self_seq_1 : [#users=1] = call_module[target=self_seq_1](args = (%self_seq_0_linear,), kwargs = {})
2022-11-23T02:25:07.1902200Z     return self_seq_1
2022-11-23T02:25:07.1902666Z [2022-11-23 02:24:53,117] torch._dynamo.optimizations.distributed: [DEBUG] run_node call_module, submod_1 got args tuple(T[torch.Size([512, 512])], T[torch.Size([512, 512])])
2022-11-23T02:25:07.1902932Z [2022-11-23 02:24:53,117] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.1903075Z ---submod_1 graph---
2022-11-23T02:25:07.1903172Z graph():
2022-11-23T02:25:07.1903422Z     %self_seq_2_weight : [#users=1] = placeholder[target=self_seq_2_weight]
2022-11-23T02:25:07.1903600Z     %self_seq_1 : [#users=1] = placeholder[target=self_seq_1]
2022-11-23T02:25:07.1903800Z     %t : [#users=1] = call_method[target=t](args = (%self_seq_2_weight,), kwargs = {})
2022-11-23T02:25:07.1904004Z     %mm : [#users=1] = call_function[target=torch.mm](args = (%self_seq_1, %t), kwargs = {})
2022-11-23T02:25:07.1904182Z     %self_seq_3 : [#users=1] = call_module[target=self_seq_3](args = (%mm,), kwargs = {})
2022-11-23T02:25:07.1904292Z     return self_seq_3
2022-11-23T02:25:07.1904716Z [2022-11-23 02:24:53,118] torch._dynamo.optimizations.distributed: [DEBUG] run_node call_module, submod_2 got args tuple(T[torch.Size([512, 512])])
2022-11-23T02:25:07.1904996Z [2022-11-23 02:24:53,118] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.1905142Z ---submod_2 graph---
2022-11-23T02:25:07.1905242Z graph():
2022-11-23T02:25:07.1905410Z     %self_seq_3 : [#users=1] = placeholder[target=self_seq_3]
2022-11-23T02:25:07.1905642Z     %self_seq_4_linear : [#users=1] = call_module[target=self_seq_4_linear](args = (%self_seq_3,), kwargs = {})
2022-11-23T02:25:07.1905844Z     %self_seq_5 : [#users=1] = call_module[target=self_seq_5](args = (%self_seq_4_linear,), kwargs = {})
2022-11-23T02:25:07.1905954Z     return self_seq_5
2022-11-23T02:25:07.1906377Z [2022-11-23 02:24:53,119] torch._dynamo.optimizations.distributed: [DEBUG] run_node output, output got args tuple(tuple(T[torch.Size([512, 512])]))
2022-11-23T02:25:07.1906653Z [2022-11-23 02:24:53,120] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.1907025Z ---final graph---
2022-11-23T02:25:07.1907271Z graph():
2022-11-23T02:25:07.1907441Z     %x : torch.Tensor [#users=1] = placeholder[target=x]
2022-11-23T02:25:07.1907623Z     %self_seq_2_weight : [#users=1] = get_attr[target=self_seq_2_weight]
2022-11-23T02:25:07.1907812Z     %submod_0 : [#users=1] = call_module[target=compiled_submod_0](args = (%x,), kwargs = {})
2022-11-23T02:25:07.1908060Z     %submod_1 : [#users=1] = call_module[target=compiled_submod_1](args = (%self_seq_2_weight, %submod_0), kwargs = {})
2022-11-23T02:25:07.1908278Z     %submod_2 : [#users=1] = call_module[target=compiled_submod_2](args = (%submod_1,), kwargs = {})
2022-11-23T02:25:07.1908390Z     return (submod_2,)
2022-11-23T02:25:07.1908538Z ---------------
2022-11-23T02:25:07.1908558Z 
2022-11-23T02:25:07.1908883Z [2022-11-23 02:24:53,120] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
2022-11-23T02:25:07.1909141Z [2022-11-23 02:24:53,120] torch._dynamo.output_graph: [CODE] TRACED GRAPH
2022-11-23T02:25:07.1909327Z  __compiled_fn_1 <eval_with_key>.26 opcode, name, target, args, kwargs
2022-11-23T02:25:07.1909435Z placeholder, x, x, (), {}
2022-11-23T02:25:07.1909605Z call_module, self_seq_0_linear, self_seq_0_linear, (x,), {}
2022-11-23T02:25:07.1909773Z call_module, self_seq_1, self_seq_1, (self_seq_0_linear,), {}
2022-11-23T02:25:07.1909931Z get_attr, self_seq_2_weight, self_seq_2_weight, (), {}
2022-11-23T02:25:07.1910077Z call_method, t, t, (self_seq_2_weight,), {}
2022-11-23T02:25:07.1910383Z call_function, mm, <built-in method mm of type object at 0x7f9905f08c60>, (self_seq_1, t), {}
2022-11-23T02:25:07.1910529Z call_module, self_seq_3, self_seq_3, (mm,), {}
2022-11-23T02:25:07.1910693Z call_module, self_seq_4_linear, self_seq_4_linear, (self_seq_3,), {}
2022-11-23T02:25:07.1910861Z call_module, self_seq_5, self_seq_5, (self_seq_4_linear,), {}
2022-11-23T02:25:07.1911009Z output, output, output, ((self_seq_5,),), {}
2022-11-23T02:25:07.1911029Z 
2022-11-23T02:25:07.1911485Z [2022-11-23 02:24:53,121] torch._dynamo.convert_frame: [CODE] ORIGINAL BYTECODE forward /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 91 
2022-11-23T02:25:07.1911621Z  92           0 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1911755Z               2 LOAD_METHOD              0 (seq)
2022-11-23T02:25:07.1911883Z               4 LOAD_FAST                1 (x)
2022-11-23T02:25:07.1912082Z               6 CALL_METHOD              1
2022-11-23T02:25:07.1912187Z               8 RETURN_VALUE
2022-11-23T02:25:07.1912225Z 
2022-11-23T02:25:07.1912299Z  
2022-11-23T02:25:07.1912759Z [2022-11-23 02:24:53,121] torch._dynamo.convert_frame: [CODE] MODIFIED BYTECODE forward /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 91 
2022-11-23T02:25:07.1912914Z  91           0 LOAD_GLOBAL              1 (__compiled_fn_1)
2022-11-23T02:25:07.1913042Z               2 LOAD_FAST                1 (x)
2022-11-23T02:25:07.1913168Z               4 CALL_FUNCTION            1
2022-11-23T02:25:07.1913300Z               6 UNPACK_SEQUENCE          1
2022-11-23T02:25:07.1913413Z               8 RETURN_VALUE
2022-11-23T02:25:07.1913432Z 
2022-11-23T02:25:07.1913505Z  
2022-11-23T02:25:07.1913756Z [2022-11-23 02:24:53,121] torch._dynamo.convert_frame: [CODE] GUARDS:
2022-11-23T02:25:07.1913858Z  - 
2022-11-23T02:25:07.1914022Z             local 'x' TENSOR_MATCH
2022-11-23T02:25:07.1914117Z             {
2022-11-23T02:25:07.1914316Z                 'guard_types': ['TENSOR_MATCH'],
2022-11-23T02:25:07.1914466Z                 'code': None,
2022-11-23T02:25:07.1914738Z                 'obj_weakref': <weakref at 0x7f98ad8b6160; to 'Tensor' at 0x7f98ad892f70>
2022-11-23T02:25:07.1915071Z                 'guarded_class': <weakref at 0x7f98dc435d50; to 'torch._C._TensorMeta' at 0x55a4248e47a0 (Tensor)>
2022-11-23T02:25:07.1915167Z             }
2022-11-23T02:25:07.1915260Z             
2022-11-23T02:25:07.1915363Z  - 
2022-11-23T02:25:07.1915526Z             local 'self' NN_MODULE
2022-11-23T02:25:07.1915620Z             {
2022-11-23T02:25:07.1915856Z                 'guard_types': ['ID_MATCH'],
2022-11-23T02:25:07.1916086Z                 'code': ['___check_obj_id(self, 140293723502480)'],
2022-11-23T02:25:07.1916375Z                 'obj_weakref': <weakref at 0x7f98ad797060; to 'MyModule' at 0x7f98ad8db790>
2022-11-23T02:25:07.1916680Z                 'guarded_class': <weakref at 0x7f98d1511350; to 'type' at 0x55a463607470 (MyModule)>
2022-11-23T02:25:07.1916780Z             }
2022-11-23T02:25:07.1916876Z             
2022-11-23T02:25:07.1916978Z  - 
2022-11-23T02:25:07.1917144Z             global 'torch' FUNCTION_MATCH
2022-11-23T02:25:07.1917238Z             {
2022-11-23T02:25:07.1917405Z                 'guard_types': None,
2022-11-23T02:25:07.1917553Z                 'code': None,
2022-11-23T02:25:07.1917714Z                 'obj_weakref': None
2022-11-23T02:25:07.1917879Z                 'guarded_class': None
2022-11-23T02:25:07.1917973Z             }
2022-11-23T02:25:07.1918049Z             
2022-11-23T02:25:07.1918151Z  - 
2022-11-23T02:25:07.1918355Z             local_nn_module 'self.seq' NN_MODULE
2022-11-23T02:25:07.1918448Z             {
2022-11-23T02:25:07.1918612Z                 'guard_types': None,
2022-11-23T02:25:07.1918759Z                 'code': None,
2022-11-23T02:25:07.1918919Z                 'obj_weakref': None
2022-11-23T02:25:07.1919067Z                 'guarded_class': None
2022-11-23T02:25:07.1919160Z             }
2022-11-23T02:25:07.1919257Z             
2022-11-23T02:25:07.1919359Z  - 
2022-11-23T02:25:07.1919563Z             local_nn_module 'self.seq[0]' NN_MODULE
2022-11-23T02:25:07.1919658Z             {
2022-11-23T02:25:07.1919805Z                 'guard_types': None,
2022-11-23T02:25:07.1919954Z                 'code': None,
2022-11-23T02:25:07.1920113Z                 'obj_weakref': None
2022-11-23T02:25:07.1920277Z                 'guarded_class': None
2022-11-23T02:25:07.1920371Z             }
2022-11-23T02:25:07.1920464Z             
2022-11-23T02:25:07.1920563Z  - 
2022-11-23T02:25:07.1920750Z             local_nn_module 'self.seq[1]' NN_MODULE
2022-11-23T02:25:07.1920848Z             {
2022-11-23T02:25:07.1921015Z                 'guard_types': None,
2022-11-23T02:25:07.1921163Z                 'code': None,
2022-11-23T02:25:07.1921323Z                 'obj_weakref': None
2022-11-23T02:25:07.1921487Z                 'guarded_class': None
2022-11-23T02:25:07.1921580Z             }
2022-11-23T02:25:07.1921721Z             
2022-11-23T02:25:07.1921834Z  - 
2022-11-23T02:25:07.1922038Z             local_nn_module 'self.seq[2]' NN_MODULE
2022-11-23T02:25:07.1922133Z             {
2022-11-23T02:25:07.1922294Z                 'guard_types': None,
2022-11-23T02:25:07.1922441Z                 'code': None,
2022-11-23T02:25:07.1922602Z                 'obj_weakref': None
2022-11-23T02:25:07.1922750Z                 'guarded_class': None
2022-11-23T02:25:07.1922845Z             }
2022-11-23T02:25:07.1922938Z             
2022-11-23T02:25:07.1923038Z  - 
2022-11-23T02:25:07.1923243Z             local_nn_module 'self.seq[3]' NN_MODULE
2022-11-23T02:25:07.1923340Z             {
2022-11-23T02:25:07.1923486Z                 'guard_types': None,
2022-11-23T02:25:07.1923635Z                 'code': None,
2022-11-23T02:25:07.1923796Z                 'obj_weakref': None
2022-11-23T02:25:07.1923960Z                 'guarded_class': None
2022-11-23T02:25:07.1924053Z             }
2022-11-23T02:25:07.1924146Z             
2022-11-23T02:25:07.1924249Z  - 
2022-11-23T02:25:07.1924433Z             local_nn_module 'self.seq[4]' NN_MODULE
2022-11-23T02:25:07.1924527Z             {
2022-11-23T02:25:07.1924689Z                 'guard_types': None,
2022-11-23T02:25:07.1924835Z                 'code': None,
2022-11-23T02:25:07.1924994Z                 'obj_weakref': None
2022-11-23T02:25:07.1925158Z                 'guarded_class': None
2022-11-23T02:25:07.1925252Z             }
2022-11-23T02:25:07.1925328Z             
2022-11-23T02:25:07.1925427Z  - 
2022-11-23T02:25:07.1925629Z             local_nn_module 'self.seq[5]' NN_MODULE
2022-11-23T02:25:07.1925795Z             {
2022-11-23T02:25:07.1925964Z                 'guard_types': None,
2022-11-23T02:25:07.1926110Z                 'code': None,
2022-11-23T02:25:07.1926253Z                 'obj_weakref': None
2022-11-23T02:25:07.1926418Z                 'guarded_class': None
2022-11-23T02:25:07.1926512Z             }
2022-11-23T02:25:07.1926605Z             
2022-11-23T02:25:07.1926708Z  - 
2022-11-23T02:25:07.1926940Z             local_nn_module 'self.seq[0].linear' NN_MODULE
2022-11-23T02:25:07.1927031Z             {
2022-11-23T02:25:07.1927178Z                 'guard_types': None,
2022-11-23T02:25:07.1927324Z                 'code': None,
2022-11-23T02:25:07.1927483Z                 'obj_weakref': None
2022-11-23T02:25:07.1927646Z                 'guarded_class': None
2022-11-23T02:25:07.1927739Z             }
2022-11-23T02:25:07.1927832Z             
2022-11-23T02:25:07.1927932Z  - 
2022-11-23T02:25:07.1928144Z             local_nn_module 'self.seq[2].weight' TENSOR_MATCH
2022-11-23T02:25:07.1928241Z             {
2022-11-23T02:25:07.1928404Z                 'guard_types': None,
2022-11-23T02:25:07.1928550Z                 'code': None,
2022-11-23T02:25:07.1928709Z                 'obj_weakref': None
2022-11-23T02:25:07.1928873Z                 'guarded_class': None
2022-11-23T02:25:07.1928966Z             }
2022-11-23T02:25:07.1929046Z             
2022-11-23T02:25:07.1929144Z  - 
2022-11-23T02:25:07.1929369Z             local_nn_module 'self.seq[4].linear' NN_MODULE
2022-11-23T02:25:07.1929463Z             {
2022-11-23T02:25:07.1929625Z                 'guard_types': None,
2022-11-23T02:25:07.1929771Z                 'code': None,
2022-11-23T02:25:07.1929913Z                 'obj_weakref': None
2022-11-23T02:25:07.1930077Z                 'guarded_class': None
2022-11-23T02:25:07.1930170Z             }
2022-11-23T02:25:07.1930263Z             
2022-11-23T02:25:07.1930426Z frames [('total', 4), ('ok', 4)]
2022-11-23T02:25:07.1930728Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2022-11-23T02:25:07.1930845Z unimplemented []
2022-11-23T02:25:07.1931129Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2022-11-23T02:25:07.1931397Z stats [('calls_captured', 8), ('fusions_possible', 7), ('unique_graphs', 1)]
2022-11-23T02:25:07.1931574Z aot_autograd [('total', 3), ('ok', 3)]
2022-11-23T02:25:07.1931737Z frames [('total', 2), ('ok', 2)]
2022-11-23T02:25:07.1932089Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2022-11-23T02:25:07.1932208Z unimplemented []
2022-11-23T02:25:07.1932511Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2022-11-23T02:25:07.1932775Z stats [('calls_captured', 7), ('fusions_possible', 6), ('unique_graphs', 1)]
2022-11-23T02:25:07.1932857Z ok (0.069s)
2022-11-23T02:25:07.1933313Z   test_ddp_baseline_aot_eager (__main__.TestDistributed) ... [2022-11-23 02:24:53,618] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
2022-11-23T02:25:07.1933750Z [2022-11-23 02:24:53,618] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:54
2022-11-23T02:25:07.1934038Z [2022-11-23 02:24:53,618] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
2022-11-23T02:25:07.1934367Z [2022-11-23 02:24:53,618] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR net [NNModuleVariable()]
2022-11-23T02:25:07.1934705Z [2022-11-23 02:24:53,619] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs [NNModuleVariable()]
2022-11-23T02:25:07.1935074Z [2022-11-23 02:24:53,619] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
2022-11-23T02:25:07.1935401Z [2022-11-23 02:24:53,654] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
2022-11-23T02:25:07.1935722Z [2022-11-23 02:24:53,654] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward
2022-11-23T02:25:07.1936093Z [2022-11-23 02:24:53,656] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
2022-11-23T02:25:07.1936410Z [2022-11-23 02:24:53,686] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
2022-11-23T02:25:07.1936666Z [2022-11-23 02:24:53,687] torch._dynamo.output_graph: [CODE] TRACED GRAPH
2022-11-23T02:25:07.1936857Z  __compiled_fn_2 <eval_with_key>.36 opcode, name, target, args, kwargs
2022-11-23T02:25:07.1936997Z placeholder, inputs, inputs, (), {}
2022-11-23T02:25:07.1937152Z call_module, self_net_0, self_net_0, (inputs,), {}
2022-11-23T02:25:07.1937309Z call_module, self_net_1, self_net_1, (self_net_0,), {}
2022-11-23T02:25:07.1937467Z call_module, self_net_2, self_net_2, (self_net_1,), {}
2022-11-23T02:25:07.1937601Z call_module, self_net_3, self_net_3, (self_net_2,), {}
2022-11-23T02:25:07.1937756Z call_module, self_net_4, self_net_4, (self_net_3,), {}
2022-11-23T02:25:07.1937908Z call_module, self_net_5, self_net_5, (self_net_4,), {}
2022-11-23T02:25:07.1938064Z call_module, self_net_6, self_net_6, (self_net_5,), {}
2022-11-23T02:25:07.1938215Z call_module, self_net_7, self_net_7, (self_net_6,), {}
2022-11-23T02:25:07.1938364Z output, output, output, ((self_net_7,),), {}
2022-11-23T02:25:07.1938385Z 
2022-11-23T02:25:07.1938849Z [2022-11-23 02:24:53,687] torch._dynamo.convert_frame: [CODE] ORIGINAL BYTECODE forward /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 53 
2022-11-23T02:25:07.1938984Z  54           0 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1939102Z               2 LOAD_METHOD              0 (net)
2022-11-23T02:25:07.1939236Z               4 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.1939361Z               6 CALL_METHOD              1
2022-11-23T02:25:07.1939473Z               8 RETURN_VALUE
2022-11-23T02:25:07.1939493Z 
2022-11-23T02:25:07.1939585Z  
2022-11-23T02:25:07.1940039Z [2022-11-23 02:24:53,687] torch._dynamo.convert_frame: [CODE] MODIFIED BYTECODE forward /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 53 
2022-11-23T02:25:07.1940195Z  53           0 LOAD_GLOBAL              1 (__compiled_fn_2)
2022-11-23T02:25:07.1940317Z               2 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.1940441Z               4 CALL_FUNCTION            1
2022-11-23T02:25:07.1940572Z               6 UNPACK_SEQUENCE          1
2022-11-23T02:25:07.1940734Z               8 RETURN_VALUE
2022-11-23T02:25:07.1940756Z 
2022-11-23T02:25:07.1940851Z  
2022-11-23T02:25:07.1941108Z [2022-11-23 02:24:53,688] torch._dynamo.convert_frame: [CODE] GUARDS:
2022-11-23T02:25:07.1941212Z  - 
2022-11-23T02:25:07.1941376Z             local 'self' NN_MODULE
2022-11-23T02:25:07.1941455Z             {
2022-11-23T02:25:07.1941641Z                 'guard_types': ['ID_MATCH'],
2022-11-23T02:25:07.1941869Z                 'code': ['___check_obj_id(self, 140293723502576)'],
2022-11-23T02:25:07.1942156Z                 'obj_weakref': <weakref at 0x7f98ad892700; to 'ToyModel' at 0x7f98ad8db7f0>
2022-11-23T02:25:07.1942468Z                 'guarded_class': <weakref at 0x7f9a4b8d0630; to 'type' at 0x55a4264bb360 (ToyModel)>
2022-11-23T02:25:07.1942562Z             }
2022-11-23T02:25:07.1942657Z             
2022-11-23T02:25:07.1942744Z  - 
2022-11-23T02:25:07.1942918Z             local 'inputs' TENSOR_MATCH
2022-11-23T02:25:07.1943012Z             {
2022-11-23T02:25:07.1943207Z                 'guard_types': ['TENSOR_MATCH'],
2022-11-23T02:25:07.1943356Z                 'code': None,
2022-11-23T02:25:07.1943641Z                 'obj_weakref': <weakref at 0x7f98ad7be8e0; to 'Tensor' at 0x7f98ad8b7a60>
2022-11-23T02:25:07.1943976Z                 'guarded_class': <weakref at 0x7f98dc435d50; to 'torch._C._TensorMeta' at 0x55a4248e47a0 (Tensor)>
2022-11-23T02:25:07.1944054Z             }
2022-11-23T02:25:07.1944147Z             
2022-11-23T02:25:07.1944250Z  - 
2022-11-23T02:25:07.1944451Z             local_nn_module 'self.net' NN_MODULE
2022-11-23T02:25:07.1944545Z             {
2022-11-23T02:25:07.1944778Z                 'guard_types': None,
2022-11-23T02:25:07.1944927Z                 'code': None,
2022-11-23T02:25:07.1945073Z                 'obj_weakref': None
2022-11-23T02:25:07.1945240Z                 'guarded_class': None
2022-11-23T02:25:07.1945334Z             }
2022-11-23T02:25:07.1945429Z             
2022-11-23T02:25:07.1945530Z  - 
2022-11-23T02:25:07.1945741Z             local_nn_module 'self.net[0]' NN_MODULE
2022-11-23T02:25:07.1945817Z             {
2022-11-23T02:25:07.1945983Z                 'guard_types': None,
2022-11-23T02:25:07.1946130Z                 'code': None,
2022-11-23T02:25:07.1946292Z                 'obj_weakref': None
2022-11-23T02:25:07.1946454Z                 'guarded_class': None
2022-11-23T02:25:07.1946547Z             }
2022-11-23T02:25:07.1946639Z             
2022-11-23T02:25:07.1946722Z  - 
2022-11-23T02:25:07.1947170Z             local_nn_module 'self.net[1]' NN_MODULE
2022-11-23T02:25:07.1947267Z             {
2022-11-23T02:25:07.1947433Z                 'guard_types': None,
2022-11-23T02:25:07.1947588Z                 'code': None,
2022-11-23T02:25:07.1947750Z                 'obj_weakref': None
2022-11-23T02:25:07.1947916Z                 'guarded_class': None
2022-11-23T02:25:07.1947991Z             }
2022-11-23T02:25:07.1948082Z             
2022-11-23T02:25:07.1948182Z  - 
2022-11-23T02:25:07.1948386Z             local_nn_module 'self.net[2]' NN_MODULE
2022-11-23T02:25:07.1948480Z             {
2022-11-23T02:25:07.1948645Z                 'guard_types': None,
2022-11-23T02:25:07.1948775Z                 'code': None,
2022-11-23T02:25:07.1948938Z                 'obj_weakref': None
2022-11-23T02:25:07.1949103Z                 'guarded_class': None
2022-11-23T02:25:07.1949195Z             }
2022-11-23T02:25:07.1949288Z             
2022-11-23T02:25:07.1949390Z  - 
2022-11-23T02:25:07.1949594Z             local_nn_module 'self.net[3]' NN_MODULE
2022-11-23T02:25:07.1949670Z             {
2022-11-23T02:25:07.1949834Z                 'guard_types': None,
2022-11-23T02:25:07.1949984Z                 'code': None,
2022-11-23T02:25:07.1950138Z                 'obj_weakref': None
2022-11-23T02:25:07.1950298Z                 'guarded_class': None
2022-11-23T02:25:07.1950399Z             }
2022-11-23T02:25:07.1950492Z             
2022-11-23T02:25:07.1950576Z  - 
2022-11-23T02:25:07.1950776Z             local_nn_module 'self.net[4]' NN_MODULE
2022-11-23T02:25:07.1950954Z             {
2022-11-23T02:25:07.1951134Z                 'guard_types': None,
2022-11-23T02:25:07.1951282Z                 'code': None,
2022-11-23T02:25:07.1951443Z                 'obj_weakref': None
2022-11-23T02:25:07.1951607Z                 'guarded_class': None
2022-11-23T02:25:07.1951683Z             }
2022-11-23T02:25:07.1951777Z             
2022-11-23T02:25:07.1951876Z  - 
2022-11-23T02:25:07.1952075Z             local_nn_module 'self.net[5]' NN_MODULE
2022-11-23T02:25:07.1952167Z             {
2022-11-23T02:25:07.1952327Z                 'guard_types': None,
2022-11-23T02:25:07.1952473Z                 'code': None,
2022-11-23T02:25:07.1952622Z                 'obj_weakref': None
2022-11-23T02:25:07.1952784Z                 'guarded_class': None
2022-11-23T02:25:07.1952910Z             }
2022-11-23T02:25:07.1953006Z             
2022-11-23T02:25:07.1953111Z  - 
2022-11-23T02:25:07.1953314Z             local_nn_module 'self.net[6]' NN_MODULE
2022-11-23T02:25:07.1953407Z             {
2022-11-23T02:25:07.1953557Z                 'guard_types': None,
2022-11-23T02:25:07.1953703Z                 'code': None,
2022-11-23T02:25:07.1953864Z                 'obj_weakref': None
2022-11-23T02:25:07.1954029Z                 'guarded_class': None
2022-11-23T02:25:07.1954120Z             }
2022-11-23T02:25:07.1954212Z             
2022-11-23T02:25:07.1954295Z  - 
2022-11-23T02:25:07.1954497Z             local_nn_module 'self.net[7]' NN_MODULE
2022-11-23T02:25:07.1954588Z             {
2022-11-23T02:25:07.1954749Z                 'guard_types': None,
2022-11-23T02:25:07.1954895Z                 'code': None,
2022-11-23T02:25:07.1955142Z                 'obj_weakref': None
2022-11-23T02:25:07.1955307Z                 'guarded_class': None
2022-11-23T02:25:07.1955383Z             }
2022-11-23T02:25:07.1955475Z             
2022-11-23T02:25:07.1955636Z frames [('total', 1), ('ok', 1)]
2022-11-23T02:25:07.1955906Z stats [('calls_captured', 8), ('fusions_possible', 7), ('unique_graphs', 1)]
2022-11-23T02:25:07.1956086Z aot_autograd [('total', 1), ('ok', 1)]
2022-11-23T02:25:07.1956184Z ok (0.706s)
2022-11-23T02:25:07.1956452Z   test_ddp_baseline_inductor (__main__.TestDistributed) ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s)
2022-11-23T02:25:07.1956699Z   test_empty_graph_inductor (__main__.TestDistributed) ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s)
2022-11-23T02:25:07.1956855Z   test_graph_split (__main__.TestDistributed)
2022-11-23T02:25:07.1957337Z Just ensures that the appropriate number of splits happen (based on ... [2022-11-23 02:24:54,323] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing opt_fn
2022-11-23T02:25:07.1957775Z [2022-11-23 02:24:54,324] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:381
2022-11-23T02:25:07.1958068Z [2022-11-23 02:24:54,324] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_DEREF ddp_m []
2022-11-23T02:25:07.1958509Z [2022-11-23 02:24:54,324] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs [UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.1958974Z [2022-11-23 02:24:54,324] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [UnspecializedNNModuleVariable(DistributedDataParallel), TensorVariable()]
2022-11-23T02:25:07.1959503Z [2022-11-23 02:24:54,326] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7f98dbcd87c0, file "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1056> 
2022-11-23T02:25:07.1959642Z  1057           0 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1959784Z                2 LOAD_ATTR                1 (autograd)
2022-11-23T02:25:07.1959903Z                4 LOAD_ATTR                2 (profiler)
2022-11-23T02:25:07.1960144Z                6 LOAD_METHOD              3 (record_function)
2022-11-23T02:25:07.1960164Z 
2022-11-23T02:25:07.1960501Z 1058           8 LOAD_CONST               1 ('DistributedDataParallel.forward')
2022-11-23T02:25:07.1960523Z 
2022-11-23T02:25:07.1960656Z 1057          10 CALL_METHOD              1
2022-11-23T02:25:07.1960793Z               12 SETUP_WITH             147 (to 308)
2022-11-23T02:25:07.1960901Z               14 POP_TOP
2022-11-23T02:25:07.1960921Z 
2022-11-23T02:25:07.1961057Z 1060          16 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1961206Z               18 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.1961314Z               20 CALL_METHOD              0
2022-11-23T02:25:07.1961457Z               22 POP_JUMP_IF_FALSE       39 (to 78)
2022-11-23T02:25:07.1961588Z               24 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1961753Z               26 LOAD_ATTR                5 (require_backward_grad_sync)
2022-11-23T02:25:07.1961893Z               28 POP_JUMP_IF_FALSE       39 (to 78)
2022-11-23T02:25:07.1961912Z 
2022-11-23T02:25:07.1962041Z 1061          30 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1962176Z               32 LOAD_ATTR                6 (logger)
2022-11-23T02:25:07.1962307Z               34 LOAD_CONST               0 (None)
2022-11-23T02:25:07.1962411Z               36 IS_OP                    1
2022-11-23T02:25:07.1962546Z               38 POP_JUMP_IF_TRUE        22 (to 44)
2022-11-23T02:25:07.1962668Z               40 LOAD_ASSERTION_ERROR
2022-11-23T02:25:07.1962794Z               42 RAISE_VARARGS            1
2022-11-23T02:25:07.1962813Z 
2022-11-23T02:25:07.1962941Z 1062     >>   44 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1963071Z               46 LOAD_ATTR                6 (logger)
2022-11-23T02:25:07.1963295Z               48 LOAD_METHOD              7 (set_runtime_stats_and_log)
2022-11-23T02:25:07.1963423Z               50 CALL_METHOD              0
2022-11-23T02:25:07.1963510Z               52 POP_TOP
2022-11-23T02:25:07.1963529Z 
2022-11-23T02:25:07.1963659Z 1063          54 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1963759Z               56 DUP_TOP
2022-11-23T02:25:07.1963909Z               58 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.1964039Z               60 LOAD_CONST               2 (1)
2022-11-23T02:25:07.1964148Z               62 INPLACE_ADD
2022-11-23T02:25:07.1964248Z               64 ROT_TWO
2022-11-23T02:25:07.1964379Z               66 STORE_ATTR               8 (num_iterations)
2022-11-23T02:25:07.1964398Z 
2022-11-23T02:25:07.1964527Z 1064          68 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1964661Z               70 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1964817Z               72 LOAD_METHOD             10 (prepare_for_forward)
2022-11-23T02:25:07.1964946Z               74 CALL_METHOD              0
2022-11-23T02:25:07.1965048Z               76 POP_TOP
2022-11-23T02:25:07.1965067Z 
2022-11-23T02:25:07.1965198Z 1068     >>   78 LOAD_GLOBAL             11 (Join)
2022-11-23T02:25:07.1965349Z               80 LOAD_METHOD             12 (notify_join_context)
2022-11-23T02:25:07.1965464Z               82 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1965588Z               84 CALL_METHOD              1
2022-11-23T02:25:07.1965718Z               86 STORE_FAST               3 (work)
2022-11-23T02:25:07.1965737Z 
2022-11-23T02:25:07.1965862Z 1069          88 LOAD_FAST                3 (work)
2022-11-23T02:25:07.1966002Z               90 POP_JUMP_IF_FALSE       54 (to 108)
2022-11-23T02:25:07.1966021Z 
2022-11-23T02:25:07.1966144Z 1070          92 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1966275Z               94 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1966443Z               96 LOAD_METHOD             13 (_set_forward_pass_work_handle)
2022-11-23T02:25:07.1966465Z 
2022-11-23T02:25:07.1966576Z 1071          98 LOAD_FAST                3 (work)
2022-11-23T02:25:07.1966700Z              100 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1966866Z              102 LOAD_ATTR               14 (_divide_by_initial_world_size)
2022-11-23T02:25:07.1966885Z 
2022-11-23T02:25:07.1967062Z 1070         104 CALL_METHOD              2
2022-11-23T02:25:07.1967175Z              106 POP_TOP
2022-11-23T02:25:07.1967194Z 
2022-11-23T02:25:07.1967329Z 1080     >>  108 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1967476Z              110 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.1967603Z              112 CALL_METHOD              0
2022-11-23T02:25:07.1967728Z              114 POP_JUMP_IF_FALSE       71 (to 142)
2022-11-23T02:25:07.1967855Z              116 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1967988Z              118 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1968142Z              120 LOAD_METHOD             15 (_rebuild_buckets)
2022-11-23T02:25:07.1968267Z              122 CALL_METHOD              0
2022-11-23T02:25:07.1968405Z              124 POP_JUMP_IF_FALSE       71 (to 142)
2022-11-23T02:25:07.1968424Z 
2022-11-23T02:25:07.1968552Z 1081         126 LOAD_GLOBAL              6 (logger)
2022-11-23T02:25:07.1968667Z              128 LOAD_METHOD             16 (info)
2022-11-23T02:25:07.1968702Z 
2022-11-23T02:25:07.1968999Z 1082         130 LOAD_CONST               3 ('Reducer buckets have been rebuilt in this iteration.')
2022-11-23T02:25:07.1969019Z 
2022-11-23T02:25:07.1969144Z 1081         132 CALL_METHOD              1
2022-11-23T02:25:07.1969249Z              134 POP_TOP
2022-11-23T02:25:07.1969268Z 
2022-11-23T02:25:07.1969399Z 1084         136 LOAD_CONST               4 (True)
2022-11-23T02:25:07.1969528Z              138 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1969684Z              140 STORE_ATTR              17 (_has_rebuilt_buckets)
2022-11-23T02:25:07.1969760Z 
2022-11-23T02:25:07.1969902Z 1088     >>  142 LOAD_GLOBAL             18 (hasattr)
2022-11-23T02:25:07.1970030Z              144 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1970230Z              146 LOAD_CONST               5 ('buffer_hook')
2022-11-23T02:25:07.1970359Z              148 CALL_FUNCTION            2
2022-11-23T02:25:07.1970526Z              150 STORE_FAST               4 (buffer_hook_registered)
2022-11-23T02:25:07.1970546Z 
2022-11-23T02:25:07.1970674Z 1089         152 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1970836Z              154 LOAD_METHOD             19 (_check_sync_bufs_pre_fwd)
2022-11-23T02:25:07.1970962Z              156 CALL_METHOD              0
2022-11-23T02:25:07.1971102Z              158 POP_JUMP_IF_FALSE       84 (to 168)
2022-11-23T02:25:07.1971121Z 
2022-11-23T02:25:07.1971248Z 1090         160 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1971375Z              162 LOAD_METHOD             20 (_sync_buffers)
2022-11-23T02:25:07.1971503Z              164 CALL_METHOD              0
2022-11-23T02:25:07.1971604Z              166 POP_TOP
2022-11-23T02:25:07.1971623Z 
2022-11-23T02:25:07.1971749Z 1092     >>  168 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1971888Z              170 LOAD_ATTR               21 (_join_config)
2022-11-23T02:25:07.1972023Z              172 LOAD_ATTR               22 (enable)
2022-11-23T02:25:07.1972162Z              174 POP_JUMP_IF_FALSE       94 (to 188)
2022-11-23T02:25:07.1972181Z 
2022-11-23T02:25:07.1972307Z 1094         176 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1972480Z              178 LOAD_ATTR               23 (_check_global_requires_backward_grad_sync)
2022-11-23T02:25:07.1972499Z 
2022-11-23T02:25:07.1972630Z 1095         180 LOAD_CONST               6 (False)
2022-11-23T02:25:07.1972649Z 
2022-11-23T02:25:07.1972877Z 1094         182 LOAD_CONST               7 (('is_joined_rank',))
2022-11-23T02:25:07.1973006Z              184 CALL_FUNCTION_KW         1
2022-11-23T02:25:07.1973113Z              186 POP_TOP
2022-11-23T02:25:07.1973132Z 
2022-11-23T02:25:07.1973259Z 1098     >>  188 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1973408Z              190 LOAD_ATTR               24 (_run_ddp_forward)
2022-11-23T02:25:07.1973541Z              192 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.1973703Z              194 BUILD_MAP                0
2022-11-23T02:25:07.1973844Z              196 LOAD_FAST                2 (kwargs)
2022-11-23T02:25:07.1973968Z              198 DICT_MERGE               1
2022-11-23T02:25:07.1974098Z              200 CALL_FUNCTION_EX         1
2022-11-23T02:25:07.1974233Z              202 STORE_FAST               5 (output)
2022-11-23T02:25:07.1974252Z 
2022-11-23T02:25:07.1974381Z 1102         204 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1974544Z              206 LOAD_METHOD             25 (_check_sync_bufs_post_fwd)
2022-11-23T02:25:07.1974668Z              208 CALL_METHOD              0
2022-11-23T02:25:07.1974793Z              210 POP_JUMP_IF_FALSE      110 (to 220)
2022-11-23T02:25:07.1974813Z 
2022-11-23T02:25:07.1974938Z 1103         212 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1975081Z              214 LOAD_METHOD             20 (_sync_buffers)
2022-11-23T02:25:07.1975205Z              216 CALL_METHOD              0
2022-11-23T02:25:07.1975308Z              218 POP_TOP
2022-11-23T02:25:07.1975330Z 
2022-11-23T02:25:07.1975461Z 1105     >>  220 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1975608Z              222 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.1975731Z              224 CALL_METHOD              0
2022-11-23T02:25:07.1975854Z              226 POP_JUMP_IF_FALSE      144 (to 288)
2022-11-23T02:25:07.1975983Z              228 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1976146Z              230 LOAD_ATTR                5 (require_backward_grad_sync)
2022-11-23T02:25:07.1976285Z              232 POP_JUMP_IF_FALSE      144 (to 288)
2022-11-23T02:25:07.1976356Z 
2022-11-23T02:25:07.1976491Z 1106         234 LOAD_CONST               4 (True)
2022-11-23T02:25:07.1976619Z              236 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1976785Z              238 STORE_ATTR              26 (require_forward_param_sync)
2022-11-23T02:25:07.1976805Z 
2022-11-23T02:25:07.1976931Z 1112         240 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1977078Z              242 LOAD_ATTR               27 (find_unused_parameters)
2022-11-23T02:25:07.1977216Z              244 POP_JUMP_IF_FALSE      137 (to 274)
2022-11-23T02:25:07.1977341Z              246 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1977482Z              248 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.1977620Z              250 POP_JUMP_IF_TRUE       137 (to 274)
2022-11-23T02:25:07.1977639Z 
2022-11-23T02:25:07.1977766Z 1114         252 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1977898Z              254 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1978044Z              256 LOAD_METHOD             29 (prepare_for_backward)
2022-11-23T02:25:07.1978079Z 
2022-11-23T02:25:07.1978192Z 1115         258 LOAD_GLOBAL             30 (list)
2022-11-23T02:25:07.1978335Z              260 LOAD_GLOBAL             31 (_find_tensors)
2022-11-23T02:25:07.1978466Z              262 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1978595Z              264 CALL_FUNCTION            1
2022-11-23T02:25:07.1978724Z              266 CALL_FUNCTION            1
2022-11-23T02:25:07.1978743Z 
2022-11-23T02:25:07.1978866Z 1114         268 CALL_METHOD              1
2022-11-23T02:25:07.1978968Z              270 POP_TOP
2022-11-23T02:25:07.1979086Z              272 JUMP_FORWARD            10 (to 294)
2022-11-23T02:25:07.1979121Z 
2022-11-23T02:25:07.1979232Z 1118     >>  274 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1979363Z              276 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1979520Z              278 LOAD_METHOD             29 (prepare_for_backward)
2022-11-23T02:25:07.1979645Z              280 BUILD_LIST               0
2022-11-23T02:25:07.1979769Z              282 CALL_METHOD              1
2022-11-23T02:25:07.1979872Z              284 POP_TOP
2022-11-23T02:25:07.1980005Z              286 JUMP_FORWARD             3 (to 294)
2022-11-23T02:25:07.1980024Z 
2022-11-23T02:25:07.1980188Z 1120     >>  288 LOAD_CONST               6 (False)
2022-11-23T02:25:07.1980324Z              290 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1980493Z              292 STORE_ATTR              26 (require_forward_param_sync)
2022-11-23T02:25:07.1980604Z          >>  294 POP_BLOCK
2022-11-23T02:25:07.1980623Z 
2022-11-23T02:25:07.1980753Z 1057         296 LOAD_CONST               0 (None)
2022-11-23T02:25:07.1980856Z              298 DUP_TOP
2022-11-23T02:25:07.1980959Z              300 DUP_TOP
2022-11-23T02:25:07.1981069Z              302 CALL_FUNCTION            3
2022-11-23T02:25:07.1981171Z              304 POP_TOP
2022-11-23T02:25:07.1981309Z              306 JUMP_FORWARD             8 (to 324)
2022-11-23T02:25:07.1981429Z          >>  308 WITH_EXCEPT_START
2022-11-23T02:25:07.1981567Z              310 POP_JUMP_IF_TRUE       157 (to 314)
2022-11-23T02:25:07.1981687Z              312 RERAISE                  1
2022-11-23T02:25:07.1981788Z          >>  314 POP_TOP
2022-11-23T02:25:07.1981874Z              316 POP_TOP
2022-11-23T02:25:07.1981979Z              318 POP_TOP
2022-11-23T02:25:07.1982088Z              320 POP_EXCEPT
2022-11-23T02:25:07.1982188Z              322 POP_TOP
2022-11-23T02:25:07.1982207Z 
2022-11-23T02:25:07.1982334Z 1124     >>  324 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1982495Z              326 LOAD_ATTR               27 (find_unused_parameters)
2022-11-23T02:25:07.1982635Z              328 POP_JUMP_IF_FALSE      168 (to 336)
2022-11-23T02:25:07.1982760Z              330 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1982889Z              332 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.1983099Z              334 POP_JUMP_IF_FALSE      178 (to 356)
2022-11-23T02:25:07.1983119Z 
2022-11-23T02:25:07.1983248Z 1125     >>  336 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1983389Z              338 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.1983408Z 
2022-11-23T02:25:07.1983534Z 1124         340 EXTENDED_ARG             1
2022-11-23T02:25:07.1983673Z              342 POP_JUMP_IF_FALSE      265 (to 530)
2022-11-23T02:25:07.1983692Z 
2022-11-23T02:25:07.1983819Z 1125         344 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1983967Z              346 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.1984080Z              348 LOAD_CONST               2 (1)
2022-11-23T02:25:07.1984210Z              350 COMPARE_OP               2 (==)
2022-11-23T02:25:07.1984229Z 
2022-11-23T02:25:07.1984355Z 1124         352 EXTENDED_ARG             1
2022-11-23T02:25:07.1984493Z              354 POP_JUMP_IF_FALSE      265 (to 530)
2022-11-23T02:25:07.1984513Z 
2022-11-23T02:25:07.1984645Z 1128     >>  356 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1984788Z              358 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.1984808Z 
2022-11-23T02:25:07.1984935Z 1129         360 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1985082Z              362 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.1985101Z 
2022-11-23T02:25:07.1985359Z 1127         364 LOAD_CONST               8 (('static_graph', 'num_iterations'))
2022-11-23T02:25:07.1985496Z              366 BUILD_CONST_KEY_MAP      2
2022-11-23T02:25:07.1985638Z              368 STORE_FAST               6 (state_dict)
2022-11-23T02:25:07.1985657Z 
2022-11-23T02:25:07.1985818Z 1136         370 LOAD_GLOBAL             32 (_tree_flatten_with_rref)
2022-11-23T02:25:07.1985953Z              372 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1986081Z              374 CALL_FUNCTION            1
2022-11-23T02:25:07.1986100Z 
2022-11-23T02:25:07.1986228Z 1132         376 UNPACK_SEQUENCE          3
2022-11-23T02:25:07.1986250Z 
2022-11-23T02:25:07.1986404Z 1133         378 STORE_FAST               7 (output_tensor_list)
2022-11-23T02:25:07.1986423Z 
2022-11-23T02:25:07.1986560Z 1134         380 STORE_FAST               8 (treespec)
2022-11-23T02:25:07.1986579Z 
2022-11-23T02:25:07.1986709Z 1135         382 STORE_FAST               9 (output_is_rref)
2022-11-23T02:25:07.1986745Z 
2022-11-23T02:25:07.1987539Z 1137         384 LOAD_CONST               9 (<code object <listcomp> at 0x7f98dbcd8660, file "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1137>)
2022-11-23T02:25:07.1987890Z              386 LOAD_CONST              10 ('DistributedDataParallel.forward.<locals>.<listcomp>')
2022-11-23T02:25:07.1988019Z              388 MAKE_FUNCTION            0
2022-11-23T02:25:07.1988151Z              390 LOAD_GLOBAL             33 (range)
2022-11-23T02:25:07.1988282Z              392 LOAD_GLOBAL             34 (len)
2022-11-23T02:25:07.1988433Z              394 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.1988566Z              396 CALL_FUNCTION            1
2022-11-23T02:25:07.1988693Z              398 CALL_FUNCTION            1
2022-11-23T02:25:07.1988781Z              400 GET_ITER
2022-11-23T02:25:07.1988907Z              402 CALL_FUNCTION            1
2022-11-23T02:25:07.1989073Z              404 STORE_FAST              10 (output_placeholders)
2022-11-23T02:25:07.1989094Z 
2022-11-23T02:25:07.1989236Z 1140         406 LOAD_GLOBAL             35 (enumerate)
2022-11-23T02:25:07.1989388Z              408 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.1989512Z              410 CALL_FUNCTION            1
2022-11-23T02:25:07.1989614Z              412 GET_ITER
2022-11-23T02:25:07.1989730Z          >>  414 FOR_ITER                18 (to 452)
2022-11-23T02:25:07.1989858Z              416 UNPACK_SEQUENCE          2
2022-11-23T02:25:07.1989986Z              418 STORE_FAST              11 (i)
2022-11-23T02:25:07.1990118Z              420 STORE_FAST               5 (output)
2022-11-23T02:25:07.1990208Z 
2022-11-23T02:25:07.1990351Z 1141         422 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.1990489Z              424 LOAD_METHOD             36 (is_tensor)
2022-11-23T02:25:07.1990622Z              426 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1990747Z              428 CALL_METHOD              1
2022-11-23T02:25:07.1990874Z              430 POP_JUMP_IF_FALSE      225 (to 450)
2022-11-23T02:25:07.1991000Z              432 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1991130Z              434 LOAD_ATTR               37 (grad_fn)
2022-11-23T02:25:07.1991259Z              436 LOAD_CONST               0 (None)
2022-11-23T02:25:07.1991378Z              438 IS_OP                    0
2022-11-23T02:25:07.1991517Z              440 POP_JUMP_IF_FALSE      225 (to 450)
2022-11-23T02:25:07.1991535Z 
2022-11-23T02:25:07.1991662Z 1142         442 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1991805Z              444 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1991938Z              446 LOAD_FAST               11 (i)
2022-11-23T02:25:07.1992047Z              448 STORE_SUBSCR
2022-11-23T02:25:07.1992182Z          >>  450 JUMP_ABSOLUTE          207 (to 414)
2022-11-23T02:25:07.1992200Z 
2022-11-23T02:25:07.1992336Z 1149     >>  452 LOAD_GLOBAL             38 (_DDPSink)
2022-11-23T02:25:07.1992470Z              454 LOAD_ATTR               39 (apply)
2022-11-23T02:25:07.1992490Z 
2022-11-23T02:25:07.1992617Z 1150         456 LOAD_FAST                0 (self)
2022-11-23T02:25:07.1992749Z              458 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.1992769Z 
2022-11-23T02:25:07.1992890Z 1151         460 LOAD_FAST                6 (state_dict)
2022-11-23T02:25:07.1992926Z 
2022-11-23T02:25:07.1993033Z 1149         462 BUILD_LIST               2
2022-11-23T02:25:07.1993052Z 
2022-11-23T02:25:07.1993202Z 1152         464 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.1993225Z 
2022-11-23T02:25:07.1993348Z 1149         466 LIST_EXTEND              1
2022-11-23T02:25:07.1993458Z              468 LIST_TO_TUPLE
2022-11-23T02:25:07.1993588Z              470 CALL_FUNCTION_EX         0
2022-11-23T02:25:07.1993754Z              472 STORE_FAST              12 (passthrough_tensor_list)
2022-11-23T02:25:07.1993774Z 
2022-11-23T02:25:07.1993958Z 1154         474 LOAD_GLOBAL             33 (range)
2022-11-23T02:25:07.1994098Z              476 LOAD_GLOBAL             34 (len)
2022-11-23T02:25:07.1994239Z              478 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1994366Z              480 CALL_FUNCTION            1
2022-11-23T02:25:07.1994492Z              482 CALL_FUNCTION            1
2022-11-23T02:25:07.1994593Z              484 GET_ITER
2022-11-23T02:25:07.1994725Z          >>  486 FOR_ITER                15 (to 518)
2022-11-23T02:25:07.1994854Z              488 STORE_FAST              11 (i)
2022-11-23T02:25:07.1994873Z 
2022-11-23T02:25:07.1995033Z 1155         490 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1995161Z              492 LOAD_FAST               11 (i)
2022-11-23T02:25:07.1995256Z              494 BINARY_SUBSCR
2022-11-23T02:25:07.1995387Z              496 LOAD_CONST               0 (None)
2022-11-23T02:25:07.1995506Z              498 IS_OP                    0
2022-11-23T02:25:07.1995635Z              500 EXTENDED_ARG             1
2022-11-23T02:25:07.1995776Z              502 POP_JUMP_IF_FALSE      258 (to 516)
2022-11-23T02:25:07.1995796Z 
2022-11-23T02:25:07.1995960Z 1156         504 LOAD_FAST               12 (passthrough_tensor_list)
2022-11-23T02:25:07.1996086Z              506 LOAD_FAST               11 (i)
2022-11-23T02:25:07.1996182Z              508 BINARY_SUBSCR
2022-11-23T02:25:07.1996338Z              510 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1996463Z              512 LOAD_FAST               11 (i)
2022-11-23T02:25:07.1996572Z              514 STORE_SUBSCR
2022-11-23T02:25:07.1996768Z          >>  516 JUMP_ABSOLUTE          243 (to 486)
2022-11-23T02:25:07.1996787Z 
2022-11-23T02:25:07.1996951Z 1159     >>  518 LOAD_GLOBAL             40 (_tree_unflatten_with_rref)
2022-11-23T02:25:07.1996971Z 
2022-11-23T02:25:07.1997122Z 1160         520 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.1997258Z              522 LOAD_FAST                8 (treespec)
2022-11-23T02:25:07.1997392Z              524 LOAD_FAST                9 (output_is_rref)
2022-11-23T02:25:07.1997412Z 
2022-11-23T02:25:07.1997538Z 1159         526 CALL_FUNCTION            3
2022-11-23T02:25:07.1997671Z              528 STORE_FAST               5 (output)
2022-11-23T02:25:07.1997690Z 
2022-11-23T02:25:07.1997818Z 1162     >>  530 LOAD_FAST                5 (output)
2022-11-23T02:25:07.1997927Z              532 RETURN_VALUE
2022-11-23T02:25:07.1998018Z  
2022-11-23T02:25:07.1998037Z 
2022-11-23T02:25:07.1998501Z [2022-11-23 02:24:54,327] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1057
2022-11-23T02:25:07.1998802Z [2022-11-23 02:24:54,327] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
2022-11-23T02:25:07.1999293Z [2022-11-23 02:24:54,327] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR autograd [TorchVariable(<module 'torch' from '/opt/conda/lib/python3.10/site-packages/torch/__init__.py'>)]
2022-11-23T02:25:07.1999808Z [2022-11-23 02:24:54,328] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR profiler [TorchVariable(<module 'torch.autograd' from '/opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py'>)]
2022-11-23T02:25:07.2000375Z [2022-11-23 02:24:54,328] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR record_function [TorchVariable(<module 'torch.autograd.profiler' from '/opt/conda/lib/python3.10/site-packages/torch/autograd/profiler.py'>)]
2022-11-23T02:25:07.2000816Z [2022-11-23 02:24:54,328] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1058
2022-11-23T02:25:07.2001330Z [2022-11-23 02:24:54,329] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST DistributedDataParallel.forward [TorchVariable(<class 'torch.autograd.profiler.record_function'>)]
2022-11-23T02:25:07.2001823Z [2022-11-23 02:24:54,329] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1057
2022-11-23T02:25:07.2002316Z [2022-11-23 02:24:54,329] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TorchVariable(<class 'torch.autograd.profiler.record_function'>), ConstantVariable(str)]
2022-11-23T02:25:07.2002617Z [2022-11-23 02:24:54,329] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored
2022-11-23T02:25:07.2002950Z [2022-11-23 02:24:54,329] torch._dynamo.symbolic_convert: [DEBUG] TRACE SETUP_WITH 308 [NullContextVariable()]
2022-11-23T02:25:07.2003359Z [2022-11-23 02:24:54,329] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType)]
2022-11-23T02:25:07.2003800Z [2022-11-23 02:24:54,329] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1060
2022-11-23T02:25:07.2004159Z [2022-11-23 02:24:54,329] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [WithExitFunctionVariable()]
2022-11-23T02:25:07.2004725Z [2022-11-23 02:24:54,329] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR is_grad_enabled [WithExitFunctionVariable(), TorchVariable(<module 'torch' from '/opt/conda/lib/python3.10/site-packages/torch/__init__.py'>)]
2022-11-23T02:25:07.2005163Z [2022-11-23 02:24:54,330] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), TorchVariable(<built-in function is_grad_enabled>)]
2022-11-23T02:25:07.2005577Z [2022-11-23 02:24:54,330] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 78 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.2005996Z [2022-11-23 02:24:54,330] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.2006529Z [2022-11-23 02:24:54,330] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR require_backward_grad_sync [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.2006949Z [2022-11-23 02:24:54,330] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 78 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.2007386Z [2022-11-23 02:24:54,331] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1061
2022-11-23T02:25:07.2007733Z [2022-11-23 02:24:54,331] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.2008228Z [2022-11-23 02:24:54,331] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR logger [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.2008663Z [2022-11-23 02:24:54,331] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger)]
2022-11-23T02:25:07.2009137Z [2022-11-23 02:24:54,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger), ConstantVariable(NoneType)]
2022-11-23T02:25:07.2009549Z [2022-11-23 02:24:54,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 44 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.2009987Z [2022-11-23 02:24:54,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1062
2022-11-23T02:25:07.2010323Z [2022-11-23 02:24:54,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.2010813Z [2022-11-23 02:24:54,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR logger [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.2011280Z [2022-11-23 02:24:54,332] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR set_runtime_stats_and_log [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger)]
2022-11-23T02:25:07.2011782Z [2022-11-23 02:24:54,333] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), UserDefinedObjectVariable(instancemethod)]
2022-11-23T02:25:07.2012239Z [2022-11-23 02:24:54,334] torch._dynamo.convert_frame: [CODE] ORIGINAL BYTECODE opt_fn /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 379 
2022-11-23T02:25:07.2012375Z 381           0 LOAD_DEREF               0 (ddp_m)
2022-11-23T02:25:07.2012510Z               2 LOAD_FAST                0 (inputs)
2022-11-23T02:25:07.2012636Z               4 CALL_FUNCTION            1
2022-11-23T02:25:07.2012748Z               6 RETURN_VALUE
2022-11-23T02:25:07.2012772Z 
2022-11-23T02:25:07.2012844Z  
2022-11-23T02:25:07.2013297Z [2022-11-23 02:24:54,334] torch._dynamo.convert_frame: [CODE] MODIFIED BYTECODE opt_fn /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 379 
2022-11-23T02:25:07.2013430Z 379           0 LOAD_DEREF               0 (ddp_m)
2022-11-23T02:25:07.2013564Z               2 LOAD_FAST                0 (inputs)
2022-11-23T02:25:07.2013584Z 
2022-11-23T02:25:07.2013710Z 381           4 CALL_FUNCTION            1
2022-11-23T02:25:07.2013818Z               6 RETURN_VALUE
2022-11-23T02:25:07.2013837Z 
2022-11-23T02:25:07.2013924Z  
2022-11-23T02:25:07.2014172Z [2022-11-23 02:24:54,335] torch._dynamo.convert_frame: [CODE] GUARDS:
2022-11-23T02:25:07.2014258Z  - 
2022-11-23T02:25:07.2014425Z             local 'ddp_m' TYPE_MATCH
2022-11-23T02:25:07.2014520Z             {
2022-11-23T02:25:07.2014708Z                 'guard_types': ['TYPE_MATCH'],
2022-11-23T02:25:07.2014941Z                 'code': ['___check_type_id(ddp_m, 94163481642224)'],
2022-11-23T02:25:07.2015354Z                 'obj_weakref': <weakref at 0x7f98ad8b5b70; to 'DistributedDataParallel' at 0x7f98ad8db790>
2022-11-23T02:25:07.2015717Z                 'guarded_class': <weakref at 0x7f98dbce1a80; to 'ABCMeta' at 0x55a424dfc4f0 (DistributedDataParallel)>
2022-11-23T02:25:07.2015813Z             }
2022-11-23T02:25:07.2015894Z             
2022-11-23T02:25:07.2015994Z  - 
2022-11-23T02:25:07.2016172Z             local 'inputs' TENSOR_MATCH
2022-11-23T02:25:07.2016265Z             {
2022-11-23T02:25:07.2016458Z                 'guard_types': ['TENSOR_MATCH'],
2022-11-23T02:25:07.2016606Z                 'code': None,
2022-11-23T02:25:07.2016872Z                 'obj_weakref': <weakref at 0x7f98ad892610; to 'Tensor' at 0x7f98d1483060>
2022-11-23T02:25:07.2017205Z                 'guarded_class': <weakref at 0x7f98dc435d50; to 'torch._C._TensorMeta' at 0x55a4248e47a0 (Tensor)>
2022-11-23T02:25:07.2017298Z             }
2022-11-23T02:25:07.2017394Z             
2022-11-23T02:25:07.2017723Z [2022-11-23 02:24:54,337] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
2022-11-23T02:25:07.2018147Z [2022-11-23 02:24:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:54
2022-11-23T02:25:07.2018436Z [2022-11-23 02:24:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
2022-11-23T02:25:07.2018762Z [2022-11-23 02:24:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR net [NNModuleVariable()]
2022-11-23T02:25:07.2019094Z [2022-11-23 02:24:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs [NNModuleVariable()]
2022-11-23T02:25:07.2019442Z [2022-11-23 02:24:54,337] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
2022-11-23T02:25:07.2019772Z [2022-11-23 02:24:54,373] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
2022-11-23T02:25:07.2020096Z [2022-11-23 02:24:54,373] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward
2022-11-23T02:25:07.2020413Z [2022-11-23 02:24:54,374] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
2022-11-23T02:25:07.2020893Z [2022-11-23 02:24:54,374] torch._dynamo.optimizations.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
2022-11-23T02:25:07.2021332Z [2022-11-23 02:24:54,375] torch._dynamo.optimizations.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
2022-11-23T02:25:07.2021607Z [2022-11-23 02:24:54,378] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.2021750Z ---orig graph---
2022-11-23T02:25:07.2021831Z graph():
2022-11-23T02:25:07.2022018Z     %inputs : torch.Tensor [#users=1] = placeholder[target=inputs]
2022-11-23T02:25:07.2022227Z     %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {})
2022-11-23T02:25:07.2022436Z     %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {})
2022-11-23T02:25:07.2022641Z     %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {})
2022-11-23T02:25:07.2022845Z     %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {})
2022-11-23T02:25:07.2023046Z     %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {})
2022-11-23T02:25:07.2023246Z     %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {})
2022-11-23T02:25:07.2023446Z     %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {})
2022-11-23T02:25:07.2023633Z     %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {})
2022-11-23T02:25:07.2023747Z     return (self_net_7,)
2022-11-23T02:25:07.2023767Z 
2022-11-23T02:25:07.2023975Z ---split graph---
2022-11-23T02:25:07.2024073Z graph():
2022-11-23T02:25:07.2024259Z     %inputs : torch.Tensor [#users=1] = placeholder[target=inputs]
2022-11-23T02:25:07.2024461Z     %submod_0 : [#users=1] = call_module[target=submod_0](args = (%inputs,), kwargs = {})
2022-11-23T02:25:07.2024659Z     %submod_1 : [#users=1] = call_module[target=submod_1](args = (%submod_0,), kwargs = {})
2022-11-23T02:25:07.2024861Z     %submod_2 : [#users=1] = call_module[target=submod_2](args = (%submod_1,), kwargs = {})
2022-11-23T02:25:07.2024955Z     return (submod_2,)
2022-11-23T02:25:07.2024974Z 
2022-11-23T02:25:07.2025122Z ---submod_0 graph---
2022-11-23T02:25:07.2025216Z graph():
2022-11-23T02:25:07.2025395Z     %inputs : torch.Tensor [#users=1] = placeholder[target=inputs]
2022-11-23T02:25:07.2025599Z     %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {})
2022-11-23T02:25:07.2025800Z     %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {})
2022-11-23T02:25:07.2025915Z     return self_net_1
2022-11-23T02:25:07.2025934Z 
2022-11-23T02:25:07.2026082Z ---submod_1 graph---
2022-11-23T02:25:07.2026161Z graph():
2022-11-23T02:25:07.2026328Z     %self_net_1 : [#users=1] = placeholder[target=self_net_1]
2022-11-23T02:25:07.2026529Z     %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {})
2022-11-23T02:25:07.2026731Z     %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {})
2022-11-23T02:25:07.2027064Z     return self_net_3
2022-11-23T02:25:07.2027087Z 
2022-11-23T02:25:07.2027250Z ---submod_2 graph---
2022-11-23T02:25:07.2027346Z graph():
2022-11-23T02:25:07.2027499Z     %self_net_3 : [#users=1] = placeholder[target=self_net_3]
2022-11-23T02:25:07.2027703Z     %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {})
2022-11-23T02:25:07.2027899Z     %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {})
2022-11-23T02:25:07.2028106Z     %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {})
2022-11-23T02:25:07.2028302Z     %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {})
2022-11-23T02:25:07.2028410Z     return self_net_7
2022-11-23T02:25:07.2028428Z 
2022-11-23T02:25:07.2028563Z ---------------
2022-11-23T02:25:07.2028667Z 
2022-11-23T02:25:07.2029070Z [2022-11-23 02:24:54,378] torch._dynamo.optimizations.distributed: [DEBUG] run_node placeholder, inputs got args tuple()
2022-11-23T02:25:07.2029490Z [2022-11-23 02:24:54,378] torch._dynamo.optimizations.distributed: [DEBUG] run_node call_module, submod_0 got args tuple(T[torch.Size([20, 10])])
2022-11-23T02:25:07.2029750Z [2022-11-23 02:24:54,378] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.2029896Z ---submod_0 graph---
2022-11-23T02:25:07.2029994Z graph():
2022-11-23T02:25:07.2030179Z     %inputs : torch.Tensor [#users=1] = placeholder[target=inputs]
2022-11-23T02:25:07.2030387Z     %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {})
2022-11-23T02:25:07.2030592Z     %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {})
2022-11-23T02:25:07.2030702Z     return self_net_1
2022-11-23T02:25:07.2031125Z [2022-11-23 02:24:54,379] torch._dynamo.optimizations.distributed: [DEBUG] run_node call_module, submod_1 got args tuple(T[torch.Size([20, 5000])])
2022-11-23T02:25:07.2031390Z [2022-11-23 02:24:54,379] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.2031536Z ---submod_1 graph---
2022-11-23T02:25:07.2031631Z graph():
2022-11-23T02:25:07.2031800Z     %self_net_1 : [#users=1] = placeholder[target=self_net_1]
2022-11-23T02:25:07.2032000Z     %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {})
2022-11-23T02:25:07.2032202Z     %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {})
2022-11-23T02:25:07.2032386Z     return self_net_3
2022-11-23T02:25:07.2032793Z [2022-11-23 02:24:54,380] torch._dynamo.optimizations.distributed: [DEBUG] run_node call_module, submod_2 got args tuple(T[torch.Size([20, 5000])])
2022-11-23T02:25:07.2033073Z [2022-11-23 02:24:54,380] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.2033221Z ---submod_2 graph---
2022-11-23T02:25:07.2033323Z graph():
2022-11-23T02:25:07.2033493Z     %self_net_3 : [#users=1] = placeholder[target=self_net_3]
2022-11-23T02:25:07.2033697Z     %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {})
2022-11-23T02:25:07.2033898Z     %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {})
2022-11-23T02:25:07.2034096Z     %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {})
2022-11-23T02:25:07.2034278Z     %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {})
2022-11-23T02:25:07.2034390Z     return self_net_7
2022-11-23T02:25:07.2034809Z [2022-11-23 02:24:54,380] torch._dynamo.optimizations.distributed: [DEBUG] run_node output, output got args tuple(tuple(T[torch.Size([20, 5])]))
2022-11-23T02:25:07.2035086Z [2022-11-23 02:24:54,381] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.2035226Z ---final graph---
2022-11-23T02:25:07.2035325Z graph():
2022-11-23T02:25:07.2035508Z     %inputs : torch.Tensor [#users=1] = placeholder[target=inputs]
2022-11-23T02:25:07.2035725Z     %submod_0 : [#users=1] = call_module[target=compiled_submod_0](args = (%inputs,), kwargs = {})
2022-11-23T02:25:07.2035925Z     %submod_1 : [#users=1] = call_module[target=compiled_submod_1](args = (%submod_0,), kwargs = {})
2022-11-23T02:25:07.2036132Z     %submod_2 : [#users=1] = call_module[target=compiled_submod_2](args = (%submod_1,), kwargs = {})
2022-11-23T02:25:07.2036242Z     return (submod_2,)
2022-11-23T02:25:07.2036376Z ---------------
2022-11-23T02:25:07.2036395Z 
2022-11-23T02:25:07.2036719Z [2022-11-23 02:24:54,381] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
2022-11-23T02:25:07.2036979Z [2022-11-23 02:24:54,381] torch._dynamo.output_graph: [CODE] TRACED GRAPH
2022-11-23T02:25:07.2037157Z  __compiled_fn_3 <eval_with_key>.43 opcode, name, target, args, kwargs
2022-11-23T02:25:07.2037296Z placeholder, inputs, inputs, (), {}
2022-11-23T02:25:07.2037535Z call_module, self_net_0, self_net_0, (inputs,), {}
2022-11-23T02:25:07.2037703Z call_module, self_net_1, self_net_1, (self_net_0,), {}
2022-11-23T02:25:07.2037858Z call_module, self_net_2, self_net_2, (self_net_1,), {}
2022-11-23T02:25:07.2038013Z call_module, self_net_3, self_net_3, (self_net_2,), {}
2022-11-23T02:25:07.2038164Z call_module, self_net_4, self_net_4, (self_net_3,), {}
2022-11-23T02:25:07.2038318Z call_module, self_net_5, self_net_5, (self_net_4,), {}
2022-11-23T02:25:07.2038470Z call_module, self_net_6, self_net_6, (self_net_5,), {}
2022-11-23T02:25:07.2038604Z call_module, self_net_7, self_net_7, (self_net_6,), {}
2022-11-23T02:25:07.2038756Z output, output, output, ((self_net_7,),), {}
2022-11-23T02:25:07.2038776Z 
2022-11-23T02:25:07.2039238Z [2022-11-23 02:24:54,382] torch._dynamo.convert_frame: [CODE] ORIGINAL BYTECODE forward /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 53 
2022-11-23T02:25:07.2039376Z  54           0 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2039512Z               2 LOAD_METHOD              0 (net)
2022-11-23T02:25:07.2039647Z               4 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.2039774Z               6 CALL_METHOD              1
2022-11-23T02:25:07.2039884Z               8 RETURN_VALUE
2022-11-23T02:25:07.2039904Z 
2022-11-23T02:25:07.2039978Z  
2022-11-23T02:25:07.2040429Z [2022-11-23 02:24:54,382] torch._dynamo.convert_frame: [CODE] MODIFIED BYTECODE forward /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 53 
2022-11-23T02:25:07.2040579Z  53           0 LOAD_GLOBAL              1 (__compiled_fn_3)
2022-11-23T02:25:07.2040784Z               2 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.2040913Z               4 CALL_FUNCTION            1
2022-11-23T02:25:07.2041045Z               6 UNPACK_SEQUENCE          1
2022-11-23T02:25:07.2041155Z               8 RETURN_VALUE
2022-11-23T02:25:07.2041175Z 
2022-11-23T02:25:07.2041264Z  
2022-11-23T02:25:07.2041504Z [2022-11-23 02:24:54,383] torch._dynamo.convert_frame: [CODE] GUARDS:
2022-11-23T02:25:07.2041606Z  - 
2022-11-23T02:25:07.2041771Z             local 'self' NN_MODULE
2022-11-23T02:25:07.2041866Z             {
2022-11-23T02:25:07.2042050Z                 'guard_types': ['ID_MATCH'],
2022-11-23T02:25:07.2042277Z                 'code': ['___check_obj_id(self, 140293723504352)'],
2022-11-23T02:25:07.2042567Z                 'obj_weakref': <weakref at 0x7f98ad62b290; to 'ToyModel' at 0x7f98ad8dbee0>
2022-11-23T02:25:07.2042859Z                 'guarded_class': <weakref at 0x7f9a4b8d0630; to 'type' at 0x55a4264bb360 (ToyModel)>
2022-11-23T02:25:07.2042959Z             }
2022-11-23T02:25:07.2043050Z             
2022-11-23T02:25:07.2043151Z  - 
2022-11-23T02:25:07.2043326Z             local 'inputs' TENSOR_MATCH
2022-11-23T02:25:07.2043417Z             {
2022-11-23T02:25:07.2043613Z                 'guard_types': ['TENSOR_MATCH'],
2022-11-23T02:25:07.2043745Z                 'code': None,
2022-11-23T02:25:07.2044026Z                 'obj_weakref': <weakref at 0x7f98ad892610; to 'Tensor' at 0x7f98d1483060>
2022-11-23T02:25:07.2044360Z                 'guarded_class': <weakref at 0x7f98dc435d50; to 'torch._C._TensorMeta' at 0x55a4248e47a0 (Tensor)>
2022-11-23T02:25:07.2044453Z             }
2022-11-23T02:25:07.2044547Z             
2022-11-23T02:25:07.2044648Z  - 
2022-11-23T02:25:07.2044846Z             local_nn_module 'self.net' NN_MODULE
2022-11-23T02:25:07.2044922Z             {
2022-11-23T02:25:07.2045084Z                 'guard_types': None,
2022-11-23T02:25:07.2045231Z                 'code': None,
2022-11-23T02:25:07.2045396Z                 'obj_weakref': None
2022-11-23T02:25:07.2045560Z                 'guarded_class': None
2022-11-23T02:25:07.2045652Z             }
2022-11-23T02:25:07.2045746Z             
2022-11-23T02:25:07.2045832Z  - 
2022-11-23T02:25:07.2046034Z             local_nn_module 'self.net[0]' NN_MODULE
2022-11-23T02:25:07.2046127Z             {
2022-11-23T02:25:07.2046352Z                 'guard_types': None,
2022-11-23T02:25:07.2046512Z                 'code': None,
2022-11-23T02:25:07.2046674Z                 'obj_weakref': None
2022-11-23T02:25:07.2046840Z                 'guarded_class': None
2022-11-23T02:25:07.2046916Z             }
2022-11-23T02:25:07.2047009Z             
2022-11-23T02:25:07.2047110Z  - 
2022-11-23T02:25:07.2047312Z             local_nn_module 'self.net[1]' NN_MODULE
2022-11-23T02:25:07.2047406Z             {
2022-11-23T02:25:07.2047569Z                 'guard_types': None,
2022-11-23T02:25:07.2047700Z                 'code': None,
2022-11-23T02:25:07.2047859Z                 'obj_weakref': None
2022-11-23T02:25:07.2048027Z                 'guarded_class': None
2022-11-23T02:25:07.2048120Z             }
2022-11-23T02:25:07.2048212Z             
2022-11-23T02:25:07.2048312Z  - 
2022-11-23T02:25:07.2048513Z             local_nn_module 'self.net[2]' NN_MODULE
2022-11-23T02:25:07.2048589Z             {
2022-11-23T02:25:07.2048752Z                 'guard_types': None,
2022-11-23T02:25:07.2048902Z                 'code': None,
2022-11-23T02:25:07.2049063Z                 'obj_weakref': None
2022-11-23T02:25:07.2049226Z                 'guarded_class': None
2022-11-23T02:25:07.2049318Z             }
2022-11-23T02:25:07.2049409Z             
2022-11-23T02:25:07.2049495Z  - 
2022-11-23T02:25:07.2049696Z             local_nn_module 'self.net[3]' NN_MODULE
2022-11-23T02:25:07.2049789Z             {
2022-11-23T02:25:07.2049953Z                 'guard_types': None,
2022-11-23T02:25:07.2050099Z                 'code': None,
2022-11-23T02:25:07.2050258Z                 'obj_weakref': None
2022-11-23T02:25:07.2050475Z                 'guarded_class': None
2022-11-23T02:25:07.2050570Z             }
2022-11-23T02:25:07.2050662Z             
2022-11-23T02:25:07.2050762Z  - 
2022-11-23T02:25:07.2050964Z             local_nn_module 'self.net[4]' NN_MODULE
2022-11-23T02:25:07.2051056Z             {
2022-11-23T02:25:07.2051218Z                 'guard_types': None,
2022-11-23T02:25:07.2051351Z                 'code': None,
2022-11-23T02:25:07.2051510Z                 'obj_weakref': None
2022-11-23T02:25:07.2051671Z                 'guarded_class': None
2022-11-23T02:25:07.2051764Z             }
2022-11-23T02:25:07.2051856Z             
2022-11-23T02:25:07.2051955Z  - 
2022-11-23T02:25:07.2052155Z             local_nn_module 'self.net[5]' NN_MODULE
2022-11-23T02:25:07.2052231Z             {
2022-11-23T02:25:07.2052392Z                 'guard_types': None,
2022-11-23T02:25:07.2052539Z                 'code': None,
2022-11-23T02:25:07.2052701Z                 'obj_weakref': None
2022-11-23T02:25:07.2052868Z                 'guarded_class': None
2022-11-23T02:25:07.2053001Z             }
2022-11-23T02:25:07.2053078Z             
2022-11-23T02:25:07.2053182Z  - 
2022-11-23T02:25:07.2053383Z             local_nn_module 'self.net[6]' NN_MODULE
2022-11-23T02:25:07.2053478Z             {
2022-11-23T02:25:07.2053642Z                 'guard_types': None,
2022-11-23T02:25:07.2053786Z                 'code': None,
2022-11-23T02:25:07.2053950Z                 'obj_weakref': None
2022-11-23T02:25:07.2054097Z                 'guarded_class': None
2022-11-23T02:25:07.2054189Z             }
2022-11-23T02:25:07.2054281Z             
2022-11-23T02:25:07.2054380Z  - 
2022-11-23T02:25:07.2054581Z             local_nn_module 'self.net[7]' NN_MODULE
2022-11-23T02:25:07.2054672Z             {
2022-11-23T02:25:07.2054835Z                 'guard_types': None,
2022-11-23T02:25:07.2054964Z                 'code': None,
2022-11-23T02:25:07.2055122Z                 'obj_weakref': None
2022-11-23T02:25:07.2055285Z                 'guarded_class': None
2022-11-23T02:25:07.2055384Z             }
2022-11-23T02:25:07.2055478Z             
2022-11-23T02:25:07.2055639Z frames [('total', 2), ('ok', 2)]
2022-11-23T02:25:07.2055945Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2022-11-23T02:25:07.2056040Z unimplemented []
2022-11-23T02:25:07.2056403Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2022-11-23T02:25:07.2056686Z stats [('calls_captured', 8), ('fusions_possible', 7), ('unique_graphs', 1)]
2022-11-23T02:25:07.2056787Z ok (0.561s)
2022-11-23T02:25:07.2056958Z   test_graph_split_inductor (__main__.TestDistributed)
2022-11-23T02:25:07.2057204Z Same as above, but using inductor backend. ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s)
2022-11-23T02:25:07.2057372Z   test_ignored_parameters (__main__.TestDistributed)
2022-11-23T02:25:07.2057865Z Verifies ddp graph-split logic ignores parameters marked to ignore on DDP module. ... [2022-11-23 02:24:54,407] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing opt_fn
2022-11-23T02:25:07.2058302Z [2022-11-23 02:24:54,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:498
2022-11-23T02:25:07.2058596Z [2022-11-23 02:24:54,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_DEREF ddp_m []
2022-11-23T02:25:07.2059034Z [2022-11-23 02:24:54,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs [UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.2059501Z [2022-11-23 02:24:54,407] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [UnspecializedNNModuleVariable(DistributedDataParallel), TensorVariable()]
2022-11-23T02:25:07.2060026Z [2022-11-23 02:24:54,409] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7f98dbcd87c0, file "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1056> 
2022-11-23T02:25:07.2060166Z  1057           0 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.2060364Z                2 LOAD_ATTR                1 (autograd)
2022-11-23T02:25:07.2060504Z                4 LOAD_ATTR                2 (profiler)
2022-11-23T02:25:07.2060655Z                6 LOAD_METHOD              3 (record_function)
2022-11-23T02:25:07.2060675Z 
2022-11-23T02:25:07.2060947Z 1058           8 LOAD_CONST               1 ('DistributedDataParallel.forward')
2022-11-23T02:25:07.2060984Z 
2022-11-23T02:25:07.2061097Z 1057          10 CALL_METHOD              1
2022-11-23T02:25:07.2061233Z               12 SETUP_WITH             147 (to 308)
2022-11-23T02:25:07.2061340Z               14 POP_TOP
2022-11-23T02:25:07.2061360Z 
2022-11-23T02:25:07.2061495Z 1060          16 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.2061644Z               18 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.2061770Z               20 CALL_METHOD              0
2022-11-23T02:25:07.2061910Z               22 POP_JUMP_IF_FALSE       39 (to 78)
2022-11-23T02:25:07.2062026Z               24 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2062189Z               26 LOAD_ATTR                5 (require_backward_grad_sync)
2022-11-23T02:25:07.2062326Z               28 POP_JUMP_IF_FALSE       39 (to 78)
2022-11-23T02:25:07.2062345Z 
2022-11-23T02:25:07.2062472Z 1061          30 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2062605Z               32 LOAD_ATTR                6 (logger)
2022-11-23T02:25:07.2062735Z               34 LOAD_CONST               0 (None)
2022-11-23T02:25:07.2062855Z               36 IS_OP                    1
2022-11-23T02:25:07.2062993Z               38 POP_JUMP_IF_TRUE        22 (to 44)
2022-11-23T02:25:07.2063098Z               40 LOAD_ASSERTION_ERROR
2022-11-23T02:25:07.2063225Z               42 RAISE_VARARGS            1
2022-11-23T02:25:07.2063244Z 
2022-11-23T02:25:07.2063370Z 1062     >>   44 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2063500Z               46 LOAD_ATTR                6 (logger)
2022-11-23T02:25:07.2063667Z               48 LOAD_METHOD              7 (set_runtime_stats_and_log)
2022-11-23T02:25:07.2063791Z               50 CALL_METHOD              0
2022-11-23T02:25:07.2063895Z               52 POP_TOP
2022-11-23T02:25:07.2063915Z 
2022-11-23T02:25:07.2064026Z 1063          54 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2064127Z               56 DUP_TOP
2022-11-23T02:25:07.2064328Z               58 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.2064469Z               60 LOAD_CONST               2 (1)
2022-11-23T02:25:07.2064582Z               62 INPLACE_ADD
2022-11-23T02:25:07.2064684Z               64 ROT_TWO
2022-11-23T02:25:07.2064832Z               66 STORE_ATTR               8 (num_iterations)
2022-11-23T02:25:07.2064852Z 
2022-11-23T02:25:07.2064982Z 1064          68 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2065100Z               70 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.2065256Z               72 LOAD_METHOD             10 (prepare_for_forward)
2022-11-23T02:25:07.2065386Z               74 CALL_METHOD              0
2022-11-23T02:25:07.2065489Z               76 POP_TOP
2022-11-23T02:25:07.2065508Z 
2022-11-23T02:25:07.2065639Z 1068     >>   78 LOAD_GLOBAL             11 (Join)
2022-11-23T02:25:07.2065791Z               80 LOAD_METHOD             12 (notify_join_context)
2022-11-23T02:25:07.2065922Z               82 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2066030Z               84 CALL_METHOD              1
2022-11-23T02:25:07.2066160Z               86 STORE_FAST               3 (work)
2022-11-23T02:25:07.2066180Z 
2022-11-23T02:25:07.2066303Z 1069          88 LOAD_FAST                3 (work)
2022-11-23T02:25:07.2066444Z               90 POP_JUMP_IF_FALSE       54 (to 108)
2022-11-23T02:25:07.2066463Z 
2022-11-23T02:25:07.2066587Z 1070          92 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2066718Z               94 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.2067198Z               96 LOAD_METHOD             13 (_set_forward_pass_work_handle)
2022-11-23T02:25:07.2067291Z 
2022-11-23T02:25:07.2067435Z 1071          98 LOAD_FAST                3 (work)
2022-11-23T02:25:07.2067566Z              100 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2067718Z              102 LOAD_ATTR               14 (_divide_by_initial_world_size)
2022-11-23T02:25:07.2067737Z 
2022-11-23T02:25:07.2067866Z 1070         104 CALL_METHOD              2
2022-11-23T02:25:07.2067973Z              106 POP_TOP
2022-11-23T02:25:07.2067992Z 
2022-11-23T02:25:07.2068124Z 1080     >>  108 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.2068272Z              110 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.2068396Z              112 CALL_METHOD              0
2022-11-23T02:25:07.2068537Z              114 POP_JUMP_IF_FALSE       71 (to 142)
2022-11-23T02:25:07.2068647Z              116 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2068780Z              118 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.2068931Z              120 LOAD_METHOD             15 (_rebuild_buckets)
2022-11-23T02:25:07.2069056Z              122 CALL_METHOD              0
2022-11-23T02:25:07.2069194Z              124 POP_JUMP_IF_FALSE       71 (to 142)
2022-11-23T02:25:07.2069213Z 
2022-11-23T02:25:07.2069387Z 1081         126 LOAD_GLOBAL              6 (logger)
2022-11-23T02:25:07.2069522Z              128 LOAD_METHOD             16 (info)
2022-11-23T02:25:07.2069541Z 
2022-11-23T02:25:07.2069867Z 1082         130 LOAD_CONST               3 ('Reducer buckets have been rebuilt in this iteration.')
2022-11-23T02:25:07.2069887Z 
2022-11-23T02:25:07.2070013Z 1081         132 CALL_METHOD              1
2022-11-23T02:25:07.2070101Z              134 POP_TOP
2022-11-23T02:25:07.2070119Z 
2022-11-23T02:25:07.2070252Z 1084         136 LOAD_CONST               4 (True)
2022-11-23T02:25:07.2070378Z              138 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2070534Z              140 STORE_ATTR              17 (_has_rebuilt_buckets)
2022-11-23T02:25:07.2070557Z 
2022-11-23T02:25:07.2070691Z 1088     >>  142 LOAD_GLOBAL             18 (hasattr)
2022-11-23T02:25:07.2070817Z              144 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2071031Z              146 LOAD_CONST               5 ('buffer_hook')
2022-11-23T02:25:07.2071159Z              148 CALL_FUNCTION            2
2022-11-23T02:25:07.2071382Z              150 STORE_FAST               4 (buffer_hook_registered)
2022-11-23T02:25:07.2071404Z 
2022-11-23T02:25:07.2071543Z 1089         152 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2071706Z              154 LOAD_METHOD             19 (_check_sync_bufs_pre_fwd)
2022-11-23T02:25:07.2071833Z              156 CALL_METHOD              0
2022-11-23T02:25:07.2071974Z              158 POP_JUMP_IF_FALSE       84 (to 168)
2022-11-23T02:25:07.2071994Z 
2022-11-23T02:25:07.2072123Z 1090         160 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2072269Z              162 LOAD_METHOD             20 (_sync_buffers)
2022-11-23T02:25:07.2072397Z              164 CALL_METHOD              0
2022-11-23T02:25:07.2072484Z              166 POP_TOP
2022-11-23T02:25:07.2072502Z 
2022-11-23T02:25:07.2072631Z 1092     >>  168 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2072772Z              170 LOAD_ATTR               21 (_join_config)
2022-11-23T02:25:07.2072907Z              172 LOAD_ATTR               22 (enable)
2022-11-23T02:25:07.2073044Z              174 POP_JUMP_IF_FALSE       94 (to 188)
2022-11-23T02:25:07.2073063Z 
2022-11-23T02:25:07.2073189Z 1094         176 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2073377Z              178 LOAD_ATTR               23 (_check_global_requires_backward_grad_sync)
2022-11-23T02:25:07.2073397Z 
2022-11-23T02:25:07.2073527Z 1095         180 LOAD_CONST               6 (False)
2022-11-23T02:25:07.2073546Z 
2022-11-23T02:25:07.2073761Z 1094         182 LOAD_CONST               7 (('is_joined_rank',))
2022-11-23T02:25:07.2073893Z              184 CALL_FUNCTION_KW         1
2022-11-23T02:25:07.2074057Z              186 POP_TOP
2022-11-23T02:25:07.2074075Z 
2022-11-23T02:25:07.2074205Z 1098     >>  188 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2074355Z              190 LOAD_ATTR               24 (_run_ddp_forward)
2022-11-23T02:25:07.2074489Z              192 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.2074615Z              194 BUILD_MAP                0
2022-11-23T02:25:07.2074748Z              196 LOAD_FAST                2 (kwargs)
2022-11-23T02:25:07.2074855Z              198 DICT_MERGE               1
2022-11-23T02:25:07.2074982Z              200 CALL_FUNCTION_EX         1
2022-11-23T02:25:07.2075112Z              202 STORE_FAST               5 (output)
2022-11-23T02:25:07.2075131Z 
2022-11-23T02:25:07.2075256Z 1102         204 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2075420Z              206 LOAD_METHOD             25 (_check_sync_bufs_post_fwd)
2022-11-23T02:25:07.2075544Z              208 CALL_METHOD              0
2022-11-23T02:25:07.2075685Z              210 POP_JUMP_IF_FALSE      110 (to 220)
2022-11-23T02:25:07.2075704Z 
2022-11-23T02:25:07.2075830Z 1103         212 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2075958Z              214 LOAD_METHOD             20 (_sync_buffers)
2022-11-23T02:25:07.2076084Z              216 CALL_METHOD              0
2022-11-23T02:25:07.2076189Z              218 POP_TOP
2022-11-23T02:25:07.2076208Z 
2022-11-23T02:25:07.2076340Z 1105     >>  220 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.2076487Z              222 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.2076610Z              224 CALL_METHOD              0
2022-11-23T02:25:07.2076750Z              226 POP_JUMP_IF_FALSE      144 (to 288)
2022-11-23T02:25:07.2076859Z              228 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2077026Z              230 LOAD_ATTR                5 (require_backward_grad_sync)
2022-11-23T02:25:07.2077164Z              232 POP_JUMP_IF_FALSE      144 (to 288)
2022-11-23T02:25:07.2077186Z 
2022-11-23T02:25:07.2077314Z 1106         234 LOAD_CONST               4 (True)
2022-11-23T02:25:07.2077439Z              236 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2077605Z              238 STORE_ATTR              26 (require_forward_param_sync)
2022-11-23T02:25:07.2077624Z 
2022-11-23T02:25:07.2077798Z 1112         240 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2077969Z              242 LOAD_ATTR               27 (find_unused_parameters)
2022-11-23T02:25:07.2078091Z              244 POP_JUMP_IF_FALSE      137 (to 274)
2022-11-23T02:25:07.2078216Z              246 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2078360Z              248 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.2078499Z              250 POP_JUMP_IF_TRUE       137 (to 274)
2022-11-23T02:25:07.2078518Z 
2022-11-23T02:25:07.2078644Z 1114         252 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2078776Z              254 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.2078940Z              256 LOAD_METHOD             29 (prepare_for_backward)
2022-11-23T02:25:07.2078959Z 
2022-11-23T02:25:07.2079087Z 1115         258 LOAD_GLOBAL             30 (list)
2022-11-23T02:25:07.2079215Z              260 LOAD_GLOBAL             31 (_find_tensors)
2022-11-23T02:25:07.2079347Z              262 LOAD_FAST                5 (output)
2022-11-23T02:25:07.2079477Z              264 CALL_FUNCTION            1
2022-11-23T02:25:07.2079603Z              266 CALL_FUNCTION            1
2022-11-23T02:25:07.2079622Z 
2022-11-23T02:25:07.2079745Z 1114         268 CALL_METHOD              1
2022-11-23T02:25:07.2079851Z              270 POP_TOP
2022-11-23T02:25:07.2079987Z              272 JUMP_FORWARD            10 (to 294)
2022-11-23T02:25:07.2080006Z 
2022-11-23T02:25:07.2080131Z 1118     >>  274 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2080249Z              276 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.2080406Z              278 LOAD_METHOD             29 (prepare_for_backward)
2022-11-23T02:25:07.2080585Z              280 BUILD_LIST               0
2022-11-23T02:25:07.2080710Z              282 CALL_METHOD              1
2022-11-23T02:25:07.2080814Z              284 POP_TOP
2022-11-23T02:25:07.2080947Z              286 JUMP_FORWARD             3 (to 294)
2022-11-23T02:25:07.2080966Z 
2022-11-23T02:25:07.2081099Z 1120     >>  288 LOAD_CONST               6 (False)
2022-11-23T02:25:07.2081226Z              290 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2081377Z              292 STORE_ATTR              26 (require_forward_param_sync)
2022-11-23T02:25:07.2081485Z          >>  294 POP_BLOCK
2022-11-23T02:25:07.2081505Z 
2022-11-23T02:25:07.2081634Z 1057         296 LOAD_CONST               0 (None)
2022-11-23T02:25:07.2081735Z              298 DUP_TOP
2022-11-23T02:25:07.2081835Z              300 DUP_TOP
2022-11-23T02:25:07.2081961Z              302 CALL_FUNCTION            3
2022-11-23T02:25:07.2082062Z              304 POP_TOP
2022-11-23T02:25:07.2082182Z              306 JUMP_FORWARD             8 (to 324)
2022-11-23T02:25:07.2082299Z          >>  308 WITH_EXCEPT_START
2022-11-23T02:25:07.2082440Z              310 POP_JUMP_IF_TRUE       157 (to 314)
2022-11-23T02:25:07.2082561Z              312 RERAISE                  1
2022-11-23T02:25:07.2082663Z          >>  314 POP_TOP
2022-11-23T02:25:07.2082763Z              316 POP_TOP
2022-11-23T02:25:07.2082864Z              318 POP_TOP
2022-11-23T02:25:07.2082957Z              320 POP_EXCEPT
2022-11-23T02:25:07.2083057Z              322 POP_TOP
2022-11-23T02:25:07.2083076Z 
2022-11-23T02:25:07.2083201Z 1124     >>  324 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2083362Z              326 LOAD_ATTR               27 (find_unused_parameters)
2022-11-23T02:25:07.2083501Z              328 POP_JUMP_IF_FALSE      168 (to 336)
2022-11-23T02:25:07.2083628Z              330 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2083770Z              332 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.2083895Z              334 POP_JUMP_IF_FALSE      178 (to 356)
2022-11-23T02:25:07.2083931Z 
2022-11-23T02:25:07.2084042Z 1125     >>  336 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2084184Z              338 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.2084203Z 
2022-11-23T02:25:07.2084331Z 1124         340 EXTENDED_ARG             1
2022-11-23T02:25:07.2084518Z              342 POP_JUMP_IF_FALSE      265 (to 530)
2022-11-23T02:25:07.2084539Z 
2022-11-23T02:25:07.2084672Z 1125         344 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2084818Z              346 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.2084947Z              348 LOAD_CONST               2 (1)
2022-11-23T02:25:07.2085076Z              350 COMPARE_OP               2 (==)
2022-11-23T02:25:07.2085095Z 
2022-11-23T02:25:07.2085206Z 1124         352 EXTENDED_ARG             1
2022-11-23T02:25:07.2085343Z              354 POP_JUMP_IF_FALSE      265 (to 530)
2022-11-23T02:25:07.2085365Z 
2022-11-23T02:25:07.2085490Z 1128     >>  356 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2085633Z              358 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.2085652Z 
2022-11-23T02:25:07.2085775Z 1129         360 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2085918Z              362 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.2085940Z 
2022-11-23T02:25:07.2086209Z 1127         364 LOAD_CONST               8 (('static_graph', 'num_iterations'))
2022-11-23T02:25:07.2086343Z              366 BUILD_CONST_KEY_MAP      2
2022-11-23T02:25:07.2086471Z              368 STORE_FAST               6 (state_dict)
2022-11-23T02:25:07.2086507Z 
2022-11-23T02:25:07.2086652Z 1136         370 LOAD_GLOBAL             32 (_tree_flatten_with_rref)
2022-11-23T02:25:07.2086779Z              372 LOAD_FAST                5 (output)
2022-11-23T02:25:07.2086903Z              374 CALL_FUNCTION            1
2022-11-23T02:25:07.2086922Z 
2022-11-23T02:25:07.2087110Z 1132         376 UNPACK_SEQUENCE          3
2022-11-23T02:25:07.2087129Z 
2022-11-23T02:25:07.2087281Z 1133         378 STORE_FAST               7 (output_tensor_list)
2022-11-23T02:25:07.2087300Z 
2022-11-23T02:25:07.2087439Z 1134         380 STORE_FAST               8 (treespec)
2022-11-23T02:25:07.2087457Z 
2022-11-23T02:25:07.2087600Z 1135         382 STORE_FAST               9 (output_is_rref)
2022-11-23T02:25:07.2087623Z 
2022-11-23T02:25:07.2088114Z 1137         384 LOAD_CONST               9 (<code object <listcomp> at 0x7f98dbcd8660, file "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1137>)
2022-11-23T02:25:07.2088434Z              386 LOAD_CONST              10 ('DistributedDataParallel.forward.<locals>.<listcomp>')
2022-11-23T02:25:07.2088547Z              388 MAKE_FUNCTION            0
2022-11-23T02:25:07.2088681Z              390 LOAD_GLOBAL             33 (range)
2022-11-23T02:25:07.2088812Z              392 LOAD_GLOBAL             34 (len)
2022-11-23T02:25:07.2088969Z              394 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.2089096Z              396 CALL_FUNCTION            1
2022-11-23T02:25:07.2089217Z              398 CALL_FUNCTION            1
2022-11-23T02:25:07.2089318Z              400 GET_ITER
2022-11-23T02:25:07.2089426Z              402 CALL_FUNCTION            1
2022-11-23T02:25:07.2089587Z              404 STORE_FAST              10 (output_placeholders)
2022-11-23T02:25:07.2089607Z 
2022-11-23T02:25:07.2089747Z 1140         406 LOAD_GLOBAL             35 (enumerate)
2022-11-23T02:25:07.2089899Z              408 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.2090025Z              410 CALL_FUNCTION            1
2022-11-23T02:25:07.2090124Z              412 GET_ITER
2022-11-23T02:25:07.2090255Z          >>  414 FOR_ITER                18 (to 452)
2022-11-23T02:25:07.2090382Z              416 UNPACK_SEQUENCE          2
2022-11-23T02:25:07.2090495Z              418 STORE_FAST              11 (i)
2022-11-23T02:25:07.2090631Z              420 STORE_FAST               5 (output)
2022-11-23T02:25:07.2090650Z 
2022-11-23T02:25:07.2090779Z 1141         422 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.2090915Z              424 LOAD_METHOD             36 (is_tensor)
2022-11-23T02:25:07.2091044Z              426 LOAD_FAST                5 (output)
2022-11-23T02:25:07.2091221Z              428 CALL_METHOD              1
2022-11-23T02:25:07.2091372Z              430 POP_JUMP_IF_FALSE      225 (to 450)
2022-11-23T02:25:07.2091486Z              432 LOAD_FAST                5 (output)
2022-11-23T02:25:07.2091619Z              434 LOAD_ATTR               37 (grad_fn)
2022-11-23T02:25:07.2091748Z              436 LOAD_CONST               0 (None)
2022-11-23T02:25:07.2091867Z              438 IS_OP                    0
2022-11-23T02:25:07.2092006Z              440 POP_JUMP_IF_FALSE      225 (to 450)
2022-11-23T02:25:07.2092025Z 
2022-11-23T02:25:07.2092153Z 1142         442 LOAD_FAST                5 (output)
2022-11-23T02:25:07.2092310Z              444 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.2092431Z              446 LOAD_FAST               11 (i)
2022-11-23T02:25:07.2092525Z              448 STORE_SUBSCR
2022-11-23T02:25:07.2092657Z          >>  450 JUMP_ABSOLUTE          207 (to 414)
2022-11-23T02:25:07.2092675Z 
2022-11-23T02:25:07.2092814Z 1149     >>  452 LOAD_GLOBAL             38 (_DDPSink)
2022-11-23T02:25:07.2092944Z              454 LOAD_ATTR               39 (apply)
2022-11-23T02:25:07.2092963Z 
2022-11-23T02:25:07.2093089Z 1150         456 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2093221Z              458 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.2093240Z 
2022-11-23T02:25:07.2093369Z 1151         460 LOAD_FAST                6 (state_dict)
2022-11-23T02:25:07.2093388Z 
2022-11-23T02:25:07.2093509Z 1149         462 BUILD_LIST               2
2022-11-23T02:25:07.2093527Z 
2022-11-23T02:25:07.2093679Z 1152         464 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.2093762Z 
2022-11-23T02:25:07.2093878Z 1149         466 LIST_EXTEND              1
2022-11-23T02:25:07.2093992Z              468 LIST_TO_TUPLE
2022-11-23T02:25:07.2094114Z              470 CALL_FUNCTION_EX         0
2022-11-23T02:25:07.2094279Z              472 STORE_FAST              12 (passthrough_tensor_list)
2022-11-23T02:25:07.2094298Z 
2022-11-23T02:25:07.2094429Z 1154         474 LOAD_GLOBAL             33 (range)
2022-11-23T02:25:07.2094554Z              476 LOAD_GLOBAL             34 (len)
2022-11-23T02:25:07.2094709Z              478 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.2094833Z              480 CALL_FUNCTION            1
2022-11-23T02:25:07.2094943Z              482 CALL_FUNCTION            1
2022-11-23T02:25:07.2095042Z              484 GET_ITER
2022-11-23T02:25:07.2095168Z          >>  486 FOR_ITER                15 (to 518)
2022-11-23T02:25:07.2095292Z              488 STORE_FAST              11 (i)
2022-11-23T02:25:07.2095311Z 
2022-11-23T02:25:07.2095469Z 1155         490 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.2095595Z              492 LOAD_FAST               11 (i)
2022-11-23T02:25:07.2095707Z              494 BINARY_SUBSCR
2022-11-23T02:25:07.2095821Z              496 LOAD_CONST               0 (None)
2022-11-23T02:25:07.2095937Z              498 IS_OP                    0
2022-11-23T02:25:07.2096065Z              500 EXTENDED_ARG             1
2022-11-23T02:25:07.2096205Z              502 POP_JUMP_IF_FALSE      258 (to 516)
2022-11-23T02:25:07.2096223Z 
2022-11-23T02:25:07.2096390Z 1156         504 LOAD_FAST               12 (passthrough_tensor_list)
2022-11-23T02:25:07.2096516Z              506 LOAD_FAST               11 (i)
2022-11-23T02:25:07.2096625Z              508 BINARY_SUBSCR
2022-11-23T02:25:07.2096779Z              510 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.2096890Z              512 LOAD_FAST               11 (i)
2022-11-23T02:25:07.2096998Z              514 STORE_SUBSCR
2022-11-23T02:25:07.2097131Z          >>  516 JUMP_ABSOLUTE          243 (to 486)
2022-11-23T02:25:07.2097149Z 
2022-11-23T02:25:07.2097309Z 1159     >>  518 LOAD_GLOBAL             40 (_tree_unflatten_with_rref)
2022-11-23T02:25:07.2097328Z 
2022-11-23T02:25:07.2097482Z 1160         520 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.2097669Z              522 LOAD_FAST                8 (treespec)
2022-11-23T02:25:07.2097819Z              524 LOAD_FAST                9 (output_is_rref)
2022-11-23T02:25:07.2097838Z 
2022-11-23T02:25:07.2097965Z 1159         526 CALL_FUNCTION            3
2022-11-23T02:25:07.2098082Z              528 STORE_FAST               5 (output)
2022-11-23T02:25:07.2098101Z 
2022-11-23T02:25:07.2098229Z 1162     >>  530 LOAD_FAST                5 (output)
2022-11-23T02:25:07.2098332Z              532 RETURN_VALUE
2022-11-23T02:25:07.2098420Z  
2022-11-23T02:25:07.2098439Z 
2022-11-23T02:25:07.2098901Z [2022-11-23 02:24:54,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1057
2022-11-23T02:25:07.2099200Z [2022-11-23 02:24:54,410] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
2022-11-23T02:25:07.2099688Z [2022-11-23 02:24:54,411] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR autograd [TorchVariable(<module 'torch' from '/opt/conda/lib/python3.10/site-packages/torch/__init__.py'>)]
2022-11-23T02:25:07.2100215Z [2022-11-23 02:24:54,411] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR profiler [TorchVariable(<module 'torch.autograd' from '/opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py'>)]
2022-11-23T02:25:07.2100780Z [2022-11-23 02:24:54,411] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR record_function [TorchVariable(<module 'torch.autograd.profiler' from '/opt/conda/lib/python3.10/site-packages/torch/autograd/profiler.py'>)]
2022-11-23T02:25:07.2101209Z [2022-11-23 02:24:54,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1058
2022-11-23T02:25:07.2101773Z [2022-11-23 02:24:54,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST DistributedDataParallel.forward [TorchVariable(<class 'torch.autograd.profiler.record_function'>)]
2022-11-23T02:25:07.2102210Z [2022-11-23 02:24:54,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1057
2022-11-23T02:25:07.2102692Z [2022-11-23 02:24:54,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TorchVariable(<class 'torch.autograd.profiler.record_function'>), ConstantVariable(str)]
2022-11-23T02:25:07.2102990Z [2022-11-23 02:24:54,412] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored
2022-11-23T02:25:07.2103324Z [2022-11-23 02:24:54,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE SETUP_WITH 308 [NullContextVariable()]
2022-11-23T02:25:07.2103729Z [2022-11-23 02:24:54,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType)]
2022-11-23T02:25:07.2104164Z [2022-11-23 02:24:54,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1060
2022-11-23T02:25:07.2104522Z [2022-11-23 02:24:54,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [WithExitFunctionVariable()]
2022-11-23T02:25:07.2105088Z [2022-11-23 02:24:54,412] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR is_grad_enabled [WithExitFunctionVariable(), TorchVariable(<module 'torch' from '/opt/conda/lib/python3.10/site-packages/torch/__init__.py'>)]
2022-11-23T02:25:07.2105538Z [2022-11-23 02:24:54,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), TorchVariable(<built-in function is_grad_enabled>)]
2022-11-23T02:25:07.2105951Z [2022-11-23 02:24:54,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 78 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.2106309Z [2022-11-23 02:24:54,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.2107087Z [2022-11-23 02:24:54,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR require_backward_grad_sync [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.2107546Z [2022-11-23 02:24:54,413] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 78 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.2107981Z [2022-11-23 02:24:54,414] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1061
2022-11-23T02:25:07.2108327Z [2022-11-23 02:24:54,414] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.2108823Z [2022-11-23 02:24:54,414] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR logger [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.2109258Z [2022-11-23 02:24:54,414] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger)]
2022-11-23T02:25:07.2109735Z [2022-11-23 02:24:54,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger), ConstantVariable(NoneType)]
2022-11-23T02:25:07.2110144Z [2022-11-23 02:24:54,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 44 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.2110578Z [2022-11-23 02:24:54,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1062
2022-11-23T02:25:07.2110929Z [2022-11-23 02:24:54,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.2111503Z [2022-11-23 02:24:54,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR logger [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.2111955Z [2022-11-23 02:24:54,415] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR set_runtime_stats_and_log [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger)]
2022-11-23T02:25:07.2112411Z [2022-11-23 02:24:54,416] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), UserDefinedObjectVariable(instancemethod)]
2022-11-23T02:25:07.2112860Z [2022-11-23 02:24:54,417] torch._dynamo.convert_frame: [CODE] ORIGINAL BYTECODE opt_fn /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 496 
2022-11-23T02:25:07.2112992Z 498           0 LOAD_DEREF               0 (ddp_m)
2022-11-23T02:25:07.2113123Z               2 LOAD_FAST                0 (inputs)
2022-11-23T02:25:07.2113248Z               4 CALL_FUNCTION            1
2022-11-23T02:25:07.2113362Z               6 RETURN_VALUE
2022-11-23T02:25:07.2113382Z 
2022-11-23T02:25:07.2113466Z  
2022-11-23T02:25:07.2113917Z [2022-11-23 02:24:54,417] torch._dynamo.convert_frame: [CODE] MODIFIED BYTECODE opt_fn /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 496 
2022-11-23T02:25:07.2114034Z 496           0 LOAD_DEREF               0 (ddp_m)
2022-11-23T02:25:07.2114171Z               2 LOAD_FAST                0 (inputs)
2022-11-23T02:25:07.2114190Z 
2022-11-23T02:25:07.2114315Z 498           4 CALL_FUNCTION            1
2022-11-23T02:25:07.2114418Z               6 RETURN_VALUE
2022-11-23T02:25:07.2114437Z 
2022-11-23T02:25:07.2114521Z  
2022-11-23T02:25:07.2114767Z [2022-11-23 02:24:54,417] torch._dynamo.convert_frame: [CODE] GUARDS:
2022-11-23T02:25:07.2114870Z  - 
2022-11-23T02:25:07.2115040Z             local 'ddp_m' TYPE_MATCH
2022-11-23T02:25:07.2115118Z             {
2022-11-23T02:25:07.2115304Z                 'guard_types': ['TYPE_MATCH'],
2022-11-23T02:25:07.2115537Z                 'code': ['___check_type_id(ddp_m, 94163481642224)'],
2022-11-23T02:25:07.2115872Z                 'obj_weakref': <weakref at 0x7f98ad76a980; to 'DistributedDataParallel' at 0x7f98ad7f6ec0>
2022-11-23T02:25:07.2116230Z                 'guarded_class': <weakref at 0x7f98dbce1a80; to 'ABCMeta' at 0x55a424dfc4f0 (DistributedDataParallel)>
2022-11-23T02:25:07.2116321Z             }
2022-11-23T02:25:07.2116466Z             
2022-11-23T02:25:07.2116562Z  - 
2022-11-23T02:25:07.2116738Z             local 'inputs' TENSOR_MATCH
2022-11-23T02:25:07.2116832Z             {
2022-11-23T02:25:07.2117026Z                 'guard_types': ['TENSOR_MATCH'],
2022-11-23T02:25:07.2117169Z                 'code': None,
2022-11-23T02:25:07.2117454Z                 'obj_weakref': <weakref at 0x7f98ad7bf880; to 'Tensor' at 0x7f98ad629b70>
2022-11-23T02:25:07.2117788Z                 'guarded_class': <weakref at 0x7f98dc435d50; to 'torch._C._TensorMeta' at 0x55a4248e47a0 (Tensor)>
2022-11-23T02:25:07.2117865Z             }
2022-11-23T02:25:07.2117963Z             
2022-11-23T02:25:07.2118290Z [2022-11-23 02:24:54,419] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
2022-11-23T02:25:07.2118715Z [2022-11-23 02:24:54,419] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:92
2022-11-23T02:25:07.2118996Z [2022-11-23 02:24:54,419] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
2022-11-23T02:25:07.2119321Z [2022-11-23 02:24:54,419] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR seq [NNModuleVariable()]
2022-11-23T02:25:07.2119640Z [2022-11-23 02:24:54,419] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable()]
2022-11-23T02:25:07.2120008Z [2022-11-23 02:24:54,420] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
2022-11-23T02:25:07.2120519Z [2022-11-23 02:24:54,420] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7f9a4b8cb520, file "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 77> 
2022-11-23T02:25:07.2120698Z   78           0 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2120830Z               2 LOAD_METHOD              0 (linear)
2022-11-23T02:25:07.2120958Z               4 LOAD_FAST                1 (x)
2022-11-23T02:25:07.2121084Z               6 CALL_METHOD              1
2022-11-23T02:25:07.2121193Z               8 RETURN_VALUE
2022-11-23T02:25:07.2121282Z  
2022-11-23T02:25:07.2121301Z 
2022-11-23T02:25:07.2121731Z [2022-11-23 02:24:54,420] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:78
2022-11-23T02:25:07.2122019Z [2022-11-23 02:24:54,420] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
2022-11-23T02:25:07.2122332Z [2022-11-23 02:24:54,420] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR linear [NNModuleVariable()]
2022-11-23T02:25:07.2122656Z [2022-11-23 02:24:54,421] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable()]
2022-11-23T02:25:07.2123017Z [2022-11-23 02:24:54,421] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
2022-11-23T02:25:07.2123342Z [2022-11-23 02:24:54,428] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
2022-11-23T02:25:07.2123862Z [2022-11-23 02:24:54,428] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7f9a4b8cb520, file "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 77>
2022-11-23T02:25:07.2124364Z [2022-11-23 02:24:54,430] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7f9a4b8cb260, file "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 69> 
2022-11-23T02:25:07.2124497Z   70           0 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.2124629Z               2 LOAD_METHOD              1 (mm)
2022-11-23T02:25:07.2124761Z               4 LOAD_FAST                1 (x)
2022-11-23T02:25:07.2124876Z               6 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2125002Z               8 LOAD_ATTR                2 (weight)
2022-11-23T02:25:07.2125130Z              10 LOAD_METHOD              3 (t)
2022-11-23T02:25:07.2125255Z              12 CALL_METHOD              0
2022-11-23T02:25:07.2125433Z              14 CALL_METHOD              2
2022-11-23T02:25:07.2125547Z              16 RETURN_VALUE
2022-11-23T02:25:07.2125634Z  
2022-11-23T02:25:07.2125653Z 
2022-11-23T02:25:07.2126091Z [2022-11-23 02:24:54,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:70
2022-11-23T02:25:07.2126368Z [2022-11-23 02:24:54,430] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
2022-11-23T02:25:07.2126839Z [2022-11-23 02:24:54,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR mm [TorchVariable(<module 'torch' from '/opt/conda/lib/python3.10/site-packages/torch/__init__.py'>)]
2022-11-23T02:25:07.2127260Z [2022-11-23 02:24:54,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TorchVariable(<built-in method mm of type object at 0x7f9905f08c60>)]
2022-11-23T02:25:07.2127711Z [2022-11-23 02:24:54,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TorchVariable(<built-in method mm of type object at 0x7f9905f08c60>), TensorVariable()]
2022-11-23T02:25:07.2128207Z [2022-11-23 02:24:54,431] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR weight [TorchVariable(<built-in method mm of type object at 0x7f9905f08c60>), TensorVariable(), NNModuleVariable()]
2022-11-23T02:25:07.2128684Z [2022-11-23 02:24:54,433] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR t [TorchVariable(<built-in method mm of type object at 0x7f9905f08c60>), TensorVariable(), TensorVariable()]
2022-11-23T02:25:07.2129218Z [2022-11-23 02:24:54,433] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [TorchVariable(<built-in method mm of type object at 0x7f9905f08c60>), TensorVariable(), GetAttrVariable(TensorVariable(), t)]
2022-11-23T02:25:07.2129774Z [2022-11-23 02:24:54,435] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [TorchVariable(<built-in method mm of type object at 0x7f9905f08c60>), TensorVariable(), TensorVariable()]
2022-11-23T02:25:07.2130103Z [2022-11-23 02:24:54,436] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
2022-11-23T02:25:07.2130618Z [2022-11-23 02:24:54,436] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7f9a4b8cb260, file "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 69>
2022-11-23T02:25:07.2131125Z [2022-11-23 02:24:54,438] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7f9a4b8cb520, file "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 77> 
2022-11-23T02:25:07.2131257Z   78           0 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2131380Z               2 LOAD_METHOD              0 (linear)
2022-11-23T02:25:07.2131509Z               4 LOAD_FAST                1 (x)
2022-11-23T02:25:07.2131635Z               6 CALL_METHOD              1
2022-11-23T02:25:07.2131743Z               8 RETURN_VALUE
2022-11-23T02:25:07.2131831Z  
2022-11-23T02:25:07.2131851Z 
2022-11-23T02:25:07.2132289Z [2022-11-23 02:24:54,438] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:78
2022-11-23T02:25:07.2132575Z [2022-11-23 02:24:54,438] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
2022-11-23T02:25:07.2132901Z [2022-11-23 02:24:54,438] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR linear [NNModuleVariable()]
2022-11-23T02:25:07.2133205Z [2022-11-23 02:24:54,439] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable()]
2022-11-23T02:25:07.2133569Z [2022-11-23 02:24:54,439] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
2022-11-23T02:25:07.2133903Z [2022-11-23 02:24:54,446] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
2022-11-23T02:25:07.2134470Z [2022-11-23 02:24:54,446] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object forward at 0x7f9a4b8cb520, file "/var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py", line 77>
2022-11-23T02:25:07.2134805Z [2022-11-23 02:24:54,448] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
2022-11-23T02:25:07.2135123Z [2022-11-23 02:24:54,448] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward
2022-11-23T02:25:07.2135434Z [2022-11-23 02:24:54,449] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
2022-11-23T02:25:07.2135856Z [2022-11-23 02:24:54,450] torch._dynamo.optimizations.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
2022-11-23T02:25:07.2136284Z [2022-11-23 02:24:54,450] torch._dynamo.optimizations.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
2022-11-23T02:25:07.2136563Z [2022-11-23 02:24:54,452] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.2136687Z ---orig graph---
2022-11-23T02:25:07.2136782Z graph():
2022-11-23T02:25:07.2136947Z     %x : torch.Tensor [#users=1] = placeholder[target=x]
2022-11-23T02:25:07.2137166Z     %self_seq_0_linear : [#users=1] = call_module[target=self_seq_0_linear](args = (%x,), kwargs = {})
2022-11-23T02:25:07.2137385Z     %self_seq_1 : [#users=1] = call_module[target=self_seq_1](args = (%self_seq_0_linear,), kwargs = {})
2022-11-23T02:25:07.2137566Z     %self_seq_2_weight : [#users=1] = get_attr[target=self_seq_2_weight]
2022-11-23T02:25:07.2137758Z     %t : [#users=1] = call_method[target=t](args = (%self_seq_2_weight,), kwargs = {})
2022-11-23T02:25:07.2137957Z     %mm : [#users=1] = call_function[target=torch.mm](args = (%self_seq_1, %t), kwargs = {})
2022-11-23T02:25:07.2138198Z     %self_seq_3 : [#users=1] = call_module[target=self_seq_3](args = (%mm,), kwargs = {})
2022-11-23T02:25:07.2138428Z     %self_seq_4_linear : [#users=1] = call_module[target=self_seq_4_linear](args = (%self_seq_3,), kwargs = {})
2022-11-23T02:25:07.2138648Z     %self_seq_5 : [#users=1] = call_module[target=self_seq_5](args = (%self_seq_4_linear,), kwargs = {})
2022-11-23T02:25:07.2138761Z     return (self_seq_5,)
2022-11-23T02:25:07.2138780Z 
2022-11-23T02:25:07.2138921Z ---split graph---
2022-11-23T02:25:07.2139014Z graph():
2022-11-23T02:25:07.2139181Z     %x : torch.Tensor [#users=1] = placeholder[target=x]
2022-11-23T02:25:07.2139362Z     %self_seq_2_weight : [#users=1] = get_attr[target=self_seq_2_weight]
2022-11-23T02:25:07.2139563Z     %submod_0 : [#users=1] = call_module[target=submod_0](args = (%x, %self_seq_2_weight), kwargs = {})
2022-11-23T02:25:07.2139762Z     %submod_1 : [#users=1] = call_module[target=submod_1](args = (%submod_0,), kwargs = {})
2022-11-23T02:25:07.2139873Z     return (submod_1,)
2022-11-23T02:25:07.2139892Z 
2022-11-23T02:25:07.2140040Z ---submod_0 graph---
2022-11-23T02:25:07.2140134Z graph():
2022-11-23T02:25:07.2140296Z     %x : torch.Tensor [#users=1] = placeholder[target=x]
2022-11-23T02:25:07.2140479Z     %self_seq_2_weight : [#users=1] = placeholder[target=self_seq_2_weight]
2022-11-23T02:25:07.2140683Z     %self_seq_0_linear : [#users=1] = call_module[target=self_seq_0_linear](args = (%x,), kwargs = {})
2022-11-23T02:25:07.2140903Z     %self_seq_1 : [#users=1] = call_module[target=self_seq_1](args = (%self_seq_0_linear,), kwargs = {})
2022-11-23T02:25:07.2141097Z     %t : [#users=1] = call_method[target=t](args = (%self_seq_2_weight,), kwargs = {})
2022-11-23T02:25:07.2141299Z     %mm : [#users=1] = call_function[target=torch.mm](args = (%self_seq_1, %t), kwargs = {})
2022-11-23T02:25:07.2141492Z     %self_seq_3 : [#users=1] = call_module[target=self_seq_3](args = (%mm,), kwargs = {})
2022-11-23T02:25:07.2141601Z     return self_seq_3
2022-11-23T02:25:07.2141620Z 
2022-11-23T02:25:07.2141769Z ---submod_1 graph---
2022-11-23T02:25:07.2141862Z graph():
2022-11-23T02:25:07.2142018Z     %self_seq_3 : [#users=1] = placeholder[target=self_seq_3]
2022-11-23T02:25:07.2142243Z     %self_seq_4_linear : [#users=1] = call_module[target=self_seq_4_linear](args = (%self_seq_3,), kwargs = {})
2022-11-23T02:25:07.2142507Z     %self_seq_5 : [#users=1] = call_module[target=self_seq_5](args = (%self_seq_4_linear,), kwargs = {})
2022-11-23T02:25:07.2142619Z     return self_seq_5
2022-11-23T02:25:07.2142638Z 
2022-11-23T02:25:07.2142774Z ---------------
2022-11-23T02:25:07.2142793Z 
2022-11-23T02:25:07.2143164Z [2022-11-23 02:24:54,453] torch._dynamo.optimizations.distributed: [DEBUG] run_node placeholder, x got args tuple()
2022-11-23T02:25:07.2143546Z [2022-11-23 02:24:54,453] torch._dynamo.optimizations.distributed: [DEBUG] run_node get_attr, self_seq_2_weight got args tuple()
2022-11-23T02:25:07.2144007Z [2022-11-23 02:24:54,453] torch._dynamo.optimizations.distributed: [DEBUG] run_node call_module, submod_0 got args tuple(T[torch.Size([512, 512])], T[torch.Size([512, 512])])
2022-11-23T02:25:07.2144287Z [2022-11-23 02:24:54,453] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.2144416Z ---submod_0 graph---
2022-11-23T02:25:07.2144511Z graph():
2022-11-23T02:25:07.2144679Z     %x : torch.Tensor [#users=1] = placeholder[target=x]
2022-11-23T02:25:07.2144865Z     %self_seq_2_weight : [#users=1] = placeholder[target=self_seq_2_weight]
2022-11-23T02:25:07.2145080Z     %self_seq_0_linear : [#users=1] = call_module[target=self_seq_0_linear](args = (%x,), kwargs = {})
2022-11-23T02:25:07.2145300Z     %self_seq_1 : [#users=1] = call_module[target=self_seq_1](args = (%self_seq_0_linear,), kwargs = {})
2022-11-23T02:25:07.2145494Z     %t : [#users=1] = call_method[target=t](args = (%self_seq_2_weight,), kwargs = {})
2022-11-23T02:25:07.2145693Z     %mm : [#users=1] = call_function[target=torch.mm](args = (%self_seq_1, %t), kwargs = {})
2022-11-23T02:25:07.2145934Z     %self_seq_3 : [#users=1] = call_module[target=self_seq_3](args = (%mm,), kwargs = {})
2022-11-23T02:25:07.2146045Z     return self_seq_3
2022-11-23T02:25:07.2146472Z [2022-11-23 02:24:54,454] torch._dynamo.optimizations.distributed: [DEBUG] run_node call_module, submod_1 got args tuple(T[torch.Size([512, 512])])
2022-11-23T02:25:07.2146753Z [2022-11-23 02:24:54,454] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.2147151Z ---submod_1 graph---
2022-11-23T02:25:07.2147248Z graph():
2022-11-23T02:25:07.2147422Z     %self_seq_3 : [#users=1] = placeholder[target=self_seq_3]
2022-11-23T02:25:07.2147653Z     %self_seq_4_linear : [#users=1] = call_module[target=self_seq_4_linear](args = (%self_seq_3,), kwargs = {})
2022-11-23T02:25:07.2147856Z     %self_seq_5 : [#users=1] = call_module[target=self_seq_5](args = (%self_seq_4_linear,), kwargs = {})
2022-11-23T02:25:07.2147963Z     return self_seq_5
2022-11-23T02:25:07.2148381Z [2022-11-23 02:24:54,455] torch._dynamo.optimizations.distributed: [DEBUG] run_node output, output got args tuple(tuple(T[torch.Size([512, 512])]))
2022-11-23T02:25:07.2148664Z [2022-11-23 02:24:54,455] torch._dynamo.optimizations.distributed: [DEBUG] 
2022-11-23T02:25:07.2148803Z ---final graph---
2022-11-23T02:25:07.2148898Z graph():
2022-11-23T02:25:07.2149066Z     %x : torch.Tensor [#users=1] = placeholder[target=x]
2022-11-23T02:25:07.2149230Z     %self_seq_2_weight : [#users=1] = get_attr[target=self_seq_2_weight]
2022-11-23T02:25:07.2149457Z     %submod_0 : [#users=1] = call_module[target=compiled_submod_0](args = (%x, %self_seq_2_weight), kwargs = {})
2022-11-23T02:25:07.2149670Z     %submod_1 : [#users=1] = call_module[target=compiled_submod_1](args = (%submod_0,), kwargs = {})
2022-11-23T02:25:07.2149780Z     return (submod_1,)
2022-11-23T02:25:07.2149915Z ---------------
2022-11-23T02:25:07.2149936Z 
2022-11-23T02:25:07.2150253Z [2022-11-23 02:24:54,455] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
2022-11-23T02:25:07.2150512Z [2022-11-23 02:24:54,456] torch._dynamo.output_graph: [CODE] TRACED GRAPH
2022-11-23T02:25:07.2150695Z  __compiled_fn_4 <eval_with_key>.53 opcode, name, target, args, kwargs
2022-11-23T02:25:07.2150815Z placeholder, x, x, (), {}
2022-11-23T02:25:07.2150969Z call_module, self_seq_0_linear, self_seq_0_linear, (x,), {}
2022-11-23T02:25:07.2151218Z call_module, self_seq_1, self_seq_1, (self_seq_0_linear,), {}
2022-11-23T02:25:07.2151390Z get_attr, self_seq_2_weight, self_seq_2_weight, (), {}
2022-11-23T02:25:07.2151533Z call_method, t, t, (self_seq_2_weight,), {}
2022-11-23T02:25:07.2151844Z call_function, mm, <built-in method mm of type object at 0x7f9905f08c60>, (self_seq_1, t), {}
2022-11-23T02:25:07.2151991Z call_module, self_seq_3, self_seq_3, (mm,), {}
2022-11-23T02:25:07.2152172Z call_module, self_seq_4_linear, self_seq_4_linear, (self_seq_3,), {}
2022-11-23T02:25:07.2152321Z call_module, self_seq_5, self_seq_5, (self_seq_4_linear,), {}
2022-11-23T02:25:07.2152468Z output, output, output, ((self_seq_5,),), {}
2022-11-23T02:25:07.2152488Z 
2022-11-23T02:25:07.2152973Z [2022-11-23 02:24:54,456] torch._dynamo.convert_frame: [CODE] ORIGINAL BYTECODE forward /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 91 
2022-11-23T02:25:07.2153111Z  92           0 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2153248Z               2 LOAD_METHOD              0 (seq)
2022-11-23T02:25:07.2153378Z               4 LOAD_FAST                1 (x)
2022-11-23T02:25:07.2153499Z               6 CALL_METHOD              1
2022-11-23T02:25:07.2153606Z               8 RETURN_VALUE
2022-11-23T02:25:07.2153626Z 
2022-11-23T02:25:07.2153701Z  
2022-11-23T02:25:07.2154157Z [2022-11-23 02:24:54,456] torch._dynamo.convert_frame: [CODE] MODIFIED BYTECODE forward /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 91 
2022-11-23T02:25:07.2154307Z  91           0 LOAD_GLOBAL              1 (__compiled_fn_4)
2022-11-23T02:25:07.2154523Z               2 LOAD_FAST                1 (x)
2022-11-23T02:25:07.2154648Z               4 CALL_FUNCTION            1
2022-11-23T02:25:07.2154777Z               6 UNPACK_SEQUENCE          1
2022-11-23T02:25:07.2154884Z               8 RETURN_VALUE
2022-11-23T02:25:07.2154903Z 
2022-11-23T02:25:07.2154991Z  
2022-11-23T02:25:07.2155235Z [2022-11-23 02:24:54,457] torch._dynamo.convert_frame: [CODE] GUARDS:
2022-11-23T02:25:07.2155338Z  - 
2022-11-23T02:25:07.2155499Z             local 'x' TENSOR_MATCH
2022-11-23T02:25:07.2155594Z             {
2022-11-23T02:25:07.2155793Z                 'guard_types': ['TENSOR_MATCH'],
2022-11-23T02:25:07.2155941Z                 'code': None,
2022-11-23T02:25:07.2156230Z                 'obj_weakref': <weakref at 0x7f98ad7bf880; to 'Tensor' at 0x7f98ad629b70>
2022-11-23T02:25:07.2156547Z                 'guarded_class': <weakref at 0x7f98dc435d50; to 'torch._C._TensorMeta' at 0x55a4248e47a0 (Tensor)>
2022-11-23T02:25:07.2156637Z             }
2022-11-23T02:25:07.2156734Z             
2022-11-23T02:25:07.2156838Z  - 
2022-11-23T02:25:07.2157002Z             local 'self' NN_MODULE
2022-11-23T02:25:07.2157094Z             {
2022-11-23T02:25:07.2157276Z                 'guard_types': ['ID_MATCH'],
2022-11-23T02:25:07.2157485Z                 'code': ['___check_obj_id(self, 140293722322736)'],
2022-11-23T02:25:07.2157775Z                 'obj_weakref': <weakref at 0x7f98ad795120; to 'MyModule' at 0x7f98ad7bb730>
2022-11-23T02:25:07.2158080Z                 'guarded_class': <weakref at 0x7f98d1483380; to 'type' at 0x55a463f62d60 (MyModule)>
2022-11-23T02:25:07.2158173Z             }
2022-11-23T02:25:07.2158266Z             
2022-11-23T02:25:07.2158367Z  - 
2022-11-23T02:25:07.2158548Z             global 'torch' FUNCTION_MATCH
2022-11-23T02:25:07.2158625Z             {
2022-11-23T02:25:07.2158786Z                 'guard_types': None,
2022-11-23T02:25:07.2158930Z                 'code': None,
2022-11-23T02:25:07.2159092Z                 'obj_weakref': None
2022-11-23T02:25:07.2159261Z                 'guarded_class': None
2022-11-23T02:25:07.2159349Z             }
2022-11-23T02:25:07.2159438Z             
2022-11-23T02:25:07.2159523Z  - 
2022-11-23T02:25:07.2159721Z             local_nn_module 'self.seq' NN_MODULE
2022-11-23T02:25:07.2159813Z             {
2022-11-23T02:25:07.2159976Z                 'guard_types': None,
2022-11-23T02:25:07.2160120Z                 'code': None,
2022-11-23T02:25:07.2160337Z                 'obj_weakref': None
2022-11-23T02:25:07.2160496Z                 'guarded_class': None
2022-11-23T02:25:07.2160594Z             }
2022-11-23T02:25:07.2160688Z             
2022-11-23T02:25:07.2160786Z  - 
2022-11-23T02:25:07.2160990Z             local_nn_module 'self.seq[0]' NN_MODULE
2022-11-23T02:25:07.2161082Z             {
2022-11-23T02:25:07.2161243Z                 'guard_types': None,
2022-11-23T02:25:07.2161372Z                 'code': None,
2022-11-23T02:25:07.2161532Z                 'obj_weakref': None
2022-11-23T02:25:07.2161691Z                 'guarded_class': None
2022-11-23T02:25:07.2161786Z             }
2022-11-23T02:25:07.2161877Z             
2022-11-23T02:25:07.2161977Z  - 
2022-11-23T02:25:07.2162179Z             local_nn_module 'self.seq[1]' NN_MODULE
2022-11-23T02:25:07.2162256Z             {
2022-11-23T02:25:07.2162415Z                 'guard_types': None,
2022-11-23T02:25:07.2162560Z                 'code': None,
2022-11-23T02:25:07.2162724Z                 'obj_weakref': None
2022-11-23T02:25:07.2162888Z                 'guarded_class': None
2022-11-23T02:25:07.2162975Z             }
2022-11-23T02:25:07.2163050Z             
2022-11-23T02:25:07.2163150Z  - 
2022-11-23T02:25:07.2163350Z             local_nn_module 'self.seq[2]' NN_MODULE
2022-11-23T02:25:07.2163441Z             {
2022-11-23T02:25:07.2163601Z                 'guard_types': None,
2022-11-23T02:25:07.2163744Z                 'code': None,
2022-11-23T02:25:07.2163902Z                 'obj_weakref': None
2022-11-23T02:25:07.2164046Z                 'guarded_class': None
2022-11-23T02:25:07.2164244Z             }
2022-11-23T02:25:07.2164333Z             
2022-11-23T02:25:07.2164437Z  - 
2022-11-23T02:25:07.2164639Z             local_nn_module 'self.seq[3]' NN_MODULE
2022-11-23T02:25:07.2164730Z             {
2022-11-23T02:25:07.2164891Z                 'guard_types': None,
2022-11-23T02:25:07.2165022Z                 'code': None,
2022-11-23T02:25:07.2165184Z                 'obj_weakref': None
2022-11-23T02:25:07.2165345Z                 'guarded_class': None
2022-11-23T02:25:07.2165435Z             }
2022-11-23T02:25:07.2165528Z             
2022-11-23T02:25:07.2165627Z  - 
2022-11-23T02:25:07.2165815Z             local_nn_module 'self.seq[4]' NN_MODULE
2022-11-23T02:25:07.2165906Z             {
2022-11-23T02:25:07.2166068Z                 'guard_types': None,
2022-11-23T02:25:07.2166212Z                 'code': None,
2022-11-23T02:25:07.2166371Z                 'obj_weakref': None
2022-11-23T02:25:07.2166530Z                 'guarded_class': None
2022-11-23T02:25:07.2166621Z             }
2022-11-23T02:25:07.2166701Z             
2022-11-23T02:25:07.2166801Z  - 
2022-11-23T02:25:07.2166999Z             local_nn_module 'self.seq[5]' NN_MODULE
2022-11-23T02:25:07.2167090Z             {
2022-11-23T02:25:07.2167249Z                 'guard_types': None,
2022-11-23T02:25:07.2167394Z                 'code': None,
2022-11-23T02:25:07.2167552Z                 'obj_weakref': None
2022-11-23T02:25:07.2167703Z                 'guarded_class': None
2022-11-23T02:25:07.2167794Z             }
2022-11-23T02:25:07.2167885Z             
2022-11-23T02:25:07.2167982Z  - 
2022-11-23T02:25:07.2168202Z             local_nn_module 'self.seq[0].linear' NN_MODULE
2022-11-23T02:25:07.2168294Z             {
2022-11-23T02:25:07.2168456Z                 'guard_types': None,
2022-11-23T02:25:07.2168587Z                 'code': None,
2022-11-23T02:25:07.2168745Z                 'obj_weakref': None
2022-11-23T02:25:07.2168906Z                 'guarded_class': None
2022-11-23T02:25:07.2168995Z             }
2022-11-23T02:25:07.2169088Z             
2022-11-23T02:25:07.2169186Z  - 
2022-11-23T02:25:07.2169402Z             local_nn_module 'self.seq[2].weight' TENSOR_MATCH
2022-11-23T02:25:07.2169494Z             {
2022-11-23T02:25:07.2169652Z                 'guard_types': None,
2022-11-23T02:25:07.2169799Z                 'code': None,
2022-11-23T02:25:07.2169956Z                 'obj_weakref': None
2022-11-23T02:25:07.2170173Z                 'guarded_class': None
2022-11-23T02:25:07.2170274Z             }
2022-11-23T02:25:07.2170349Z             
2022-11-23T02:25:07.2170453Z  - 
2022-11-23T02:25:07.2170670Z             local_nn_module 'self.seq[4].linear' NN_MODULE
2022-11-23T02:25:07.2170761Z             {
2022-11-23T02:25:07.2170921Z                 'guard_types': None,
2022-11-23T02:25:07.2171065Z                 'code': None,
2022-11-23T02:25:07.2171224Z                 'obj_weakref': None
2022-11-23T02:25:07.2171372Z                 'guarded_class': None
2022-11-23T02:25:07.2171462Z             }
2022-11-23T02:25:07.2171558Z             
2022-11-23T02:25:07.2171722Z frames [('total', 2), ('ok', 2)]
2022-11-23T02:25:07.2172023Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2022-11-23T02:25:07.2172132Z unimplemented []
2022-11-23T02:25:07.2172432Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2022-11-23T02:25:07.2172688Z stats [('calls_captured', 7), ('fusions_possible', 6), ('unique_graphs', 1)]
2022-11-23T02:25:07.2172782Z ok (0.066s)
2022-11-23T02:25:07.2172927Z   test_no_split (__main__.TestDistributed)
2022-11-23T02:25:07.2173407Z Ensures the DDPOptimizer returns a correct, compiled module without ... [2022-11-23 02:24:54,464] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing opt_fn
2022-11-23T02:25:07.2173842Z [2022-11-23 02:24:54,464] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:417
2022-11-23T02:25:07.2174136Z [2022-11-23 02:24:54,464] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_DEREF ddp_m []
2022-11-23T02:25:07.2174637Z [2022-11-23 02:24:54,464] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs [UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.2175102Z [2022-11-23 02:24:54,464] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [UnspecializedNNModuleVariable(DistributedDataParallel), TensorVariable()]
2022-11-23T02:25:07.2175635Z [2022-11-23 02:24:54,466] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7f98dbcd87c0, file "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1056> 
2022-11-23T02:25:07.2175756Z  1057           0 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.2175894Z                2 LOAD_ATTR                1 (autograd)
2022-11-23T02:25:07.2176031Z                4 LOAD_ATTR                2 (profiler)
2022-11-23T02:25:07.2176180Z                6 LOAD_METHOD              3 (record_function)
2022-11-23T02:25:07.2176203Z 
2022-11-23T02:25:07.2176485Z 1058           8 LOAD_CONST               1 ('DistributedDataParallel.forward')
2022-11-23T02:25:07.2176504Z 
2022-11-23T02:25:07.2176628Z 1057          10 CALL_METHOD              1
2022-11-23T02:25:07.2176765Z               12 SETUP_WITH             147 (to 308)
2022-11-23T02:25:07.2176870Z               14 POP_TOP
2022-11-23T02:25:07.2176889Z 
2022-11-23T02:25:07.2177024Z 1060          16 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.2177156Z               18 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.2177279Z               20 CALL_METHOD              0
2022-11-23T02:25:07.2177418Z               22 POP_JUMP_IF_FALSE       39 (to 78)
2022-11-23T02:25:07.2177544Z               24 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2177705Z               26 LOAD_ATTR                5 (require_backward_grad_sync)
2022-11-23T02:25:07.2177846Z               28 POP_JUMP_IF_FALSE       39 (to 78)
2022-11-23T02:25:07.2177864Z 
2022-11-23T02:25:07.2177995Z 1061          30 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2178110Z               32 LOAD_ATTR                6 (logger)
2022-11-23T02:25:07.2178236Z               34 LOAD_CONST               0 (None)
2022-11-23T02:25:07.2178351Z               36 IS_OP                    1
2022-11-23T02:25:07.2178485Z               38 POP_JUMP_IF_TRUE        22 (to 44)
2022-11-23T02:25:07.2178660Z               40 LOAD_ASSERTION_ERROR
2022-11-23T02:25:07.2178795Z               42 RAISE_VARARGS            1
2022-11-23T02:25:07.2178814Z 
2022-11-23T02:25:07.2178940Z 1062     >>   44 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2179070Z               46 LOAD_ATTR                6 (logger)
2022-11-23T02:25:07.2179218Z               48 LOAD_METHOD              7 (set_runtime_stats_and_log)
2022-11-23T02:25:07.2179341Z               50 CALL_METHOD              0
2022-11-23T02:25:07.2179442Z               52 POP_TOP
2022-11-23T02:25:07.2179461Z 
2022-11-23T02:25:07.2179589Z 1063          54 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2179691Z               56 DUP_TOP
2022-11-23T02:25:07.2179838Z               58 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.2179964Z               60 LOAD_CONST               2 (1)
2022-11-23T02:25:07.2180059Z               62 INPLACE_ADD
2022-11-23T02:25:07.2180160Z               64 ROT_TWO
2022-11-23T02:25:07.2180309Z               66 STORE_ATTR               8 (num_iterations)
2022-11-23T02:25:07.2180328Z 
2022-11-23T02:25:07.2180457Z 1064          68 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2180588Z               70 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.2180745Z               72 LOAD_METHOD             10 (prepare_for_forward)
2022-11-23T02:25:07.2180869Z               74 CALL_METHOD              0
2022-11-23T02:25:07.2180970Z               76 POP_TOP
2022-11-23T02:25:07.2180990Z 
2022-11-23T02:25:07.2181104Z 1068     >>   78 LOAD_GLOBAL             11 (Join)
2022-11-23T02:25:07.2181254Z               80 LOAD_METHOD             12 (notify_join_context)
2022-11-23T02:25:07.2181442Z               82 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2181568Z               84 CALL_METHOD              1
2022-11-23T02:25:07.2181697Z               86 STORE_FAST               3 (work)
2022-11-23T02:25:07.2181716Z 
2022-11-23T02:25:07.2181841Z 1069          88 LOAD_FAST                3 (work)
2022-11-23T02:25:07.2181987Z               90 POP_JUMP_IF_FALSE       54 (to 108)
2022-11-23T02:25:07.2182006Z 
2022-11-23T02:25:07.2182128Z 1070          92 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2182244Z               94 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.2182411Z               96 LOAD_METHOD             13 (_set_forward_pass_work_handle)
2022-11-23T02:25:07.2182431Z 
2022-11-23T02:25:07.2182556Z 1071          98 LOAD_FAST                3 (work)
2022-11-23T02:25:07.2182681Z              100 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2182847Z              102 LOAD_ATTR               14 (_divide_by_initial_world_size)
2022-11-23T02:25:07.2182869Z 
2022-11-23T02:25:07.2182992Z 1070         104 CALL_METHOD              2
2022-11-23T02:25:07.2183094Z              106 POP_TOP
2022-11-23T02:25:07.2183113Z 
2022-11-23T02:25:07.2183244Z 1080     >>  108 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.2183375Z              110 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.2183508Z              112 CALL_METHOD              0
2022-11-23T02:25:07.2183650Z              114 POP_JUMP_IF_FALSE       71 (to 142)
2022-11-23T02:25:07.2183777Z              116 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2183910Z              118 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.2184060Z              120 LOAD_METHOD             15 (_rebuild_buckets)
2022-11-23T02:25:07.2184182Z              122 CALL_METHOD              0
2022-11-23T02:25:07.2184305Z              124 POP_JUMP_IF_FALSE       71 (to 142)
2022-11-23T02:25:07.2184339Z 
2022-11-23T02:25:07.2184457Z 1081         126 LOAD_GLOBAL              6 (logger)
2022-11-23T02:25:07.2184585Z              128 LOAD_METHOD             16 (info)
2022-11-23T02:25:07.2184604Z 
2022-11-23T02:25:07.2184917Z 1082         130 LOAD_CONST               3 ('Reducer buckets have been rebuilt in this iteration.')
2022-11-23T02:25:07.2184937Z 
2022-11-23T02:25:07.2185064Z 1081         132 CALL_METHOD              1
2022-11-23T02:25:07.2185226Z              134 POP_TOP
2022-11-23T02:25:07.2185248Z 
2022-11-23T02:25:07.2185383Z 1084         136 LOAD_CONST               4 (True)
2022-11-23T02:25:07.2185510Z              138 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2185664Z              140 STORE_ATTR              17 (_has_rebuilt_buckets)
2022-11-23T02:25:07.2185683Z 
2022-11-23T02:25:07.2185801Z 1088     >>  142 LOAD_GLOBAL             18 (hasattr)
2022-11-23T02:25:07.2185926Z              144 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2186143Z              146 LOAD_CONST               5 ('buffer_hook')
2022-11-23T02:25:07.2186275Z              148 CALL_FUNCTION            2
2022-11-23T02:25:07.2186436Z              150 STORE_FAST               4 (buffer_hook_registered)
2022-11-23T02:25:07.2186455Z 
2022-11-23T02:25:07.2186581Z 1089         152 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2186743Z              154 LOAD_METHOD             19 (_check_sync_bufs_pre_fwd)
2022-11-23T02:25:07.2187047Z              156 CALL_METHOD              0
2022-11-23T02:25:07.2187176Z              158 POP_JUMP_IF_FALSE       84 (to 168)
2022-11-23T02:25:07.2187196Z 
2022-11-23T02:25:07.2187324Z 1090         160 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2187470Z              162 LOAD_METHOD             20 (_sync_buffers)
2022-11-23T02:25:07.2187596Z              164 CALL_METHOD              0
2022-11-23T02:25:07.2187700Z              166 POP_TOP
2022-11-23T02:25:07.2187720Z 
2022-11-23T02:25:07.2187848Z 1092     >>  168 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2187987Z              170 LOAD_ATTR               21 (_join_config)
2022-11-23T02:25:07.2188213Z              172 LOAD_ATTR               22 (enable)
2022-11-23T02:25:07.2188334Z              174 POP_JUMP_IF_FALSE       94 (to 188)
2022-11-23T02:25:07.2188353Z 
2022-11-23T02:25:07.2188480Z 1094         176 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2188671Z              178 LOAD_ATTR               23 (_check_global_requires_backward_grad_sync)
2022-11-23T02:25:07.2188691Z 
2022-11-23T02:25:07.2188821Z 1095         180 LOAD_CONST               6 (False)
2022-11-23T02:25:07.2188840Z 
2022-11-23T02:25:07.2189078Z 1094         182 LOAD_CONST               7 (('is_joined_rank',))
2022-11-23T02:25:07.2189209Z              184 CALL_FUNCTION_KW         1
2022-11-23T02:25:07.2189312Z              186 POP_TOP
2022-11-23T02:25:07.2189331Z 
2022-11-23T02:25:07.2189460Z 1098     >>  188 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2189612Z              190 LOAD_ATTR               24 (_run_ddp_forward)
2022-11-23T02:25:07.2189728Z              192 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.2189857Z              194 BUILD_MAP                0
2022-11-23T02:25:07.2189987Z              196 LOAD_FAST                2 (kwargs)
2022-11-23T02:25:07.2190108Z              198 DICT_MERGE               1
2022-11-23T02:25:07.2190235Z              200 CALL_FUNCTION_EX         1
2022-11-23T02:25:07.2190369Z              202 STORE_FAST               5 (output)
2022-11-23T02:25:07.2190388Z 
2022-11-23T02:25:07.2190514Z 1102         204 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2190661Z              206 LOAD_METHOD             25 (_check_sync_bufs_post_fwd)
2022-11-23T02:25:07.2190784Z              208 CALL_METHOD              0
2022-11-23T02:25:07.2190925Z              210 POP_JUMP_IF_FALSE      110 (to 220)
2022-11-23T02:25:07.2190944Z 
2022-11-23T02:25:07.2191074Z 1103         212 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2191218Z              214 LOAD_METHOD             20 (_sync_buffers)
2022-11-23T02:25:07.2191341Z              216 CALL_METHOD              0
2022-11-23T02:25:07.2191447Z              218 POP_TOP
2022-11-23T02:25:07.2191466Z 
2022-11-23T02:25:07.2191597Z 1105     >>  220 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.2191728Z              222 LOAD_METHOD              4 (is_grad_enabled)
2022-11-23T02:25:07.2191850Z              224 CALL_METHOD              0
2022-11-23T02:25:07.2192061Z              226 POP_JUMP_IF_FALSE      144 (to 288)
2022-11-23T02:25:07.2192200Z              228 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2192365Z              230 LOAD_ATTR                5 (require_backward_grad_sync)
2022-11-23T02:25:07.2192506Z              232 POP_JUMP_IF_FALSE      144 (to 288)
2022-11-23T02:25:07.2192525Z 
2022-11-23T02:25:07.2192652Z 1106         234 LOAD_CONST               4 (True)
2022-11-23T02:25:07.2192778Z              236 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2192927Z              238 STORE_ATTR              26 (require_forward_param_sync)
2022-11-23T02:25:07.2192950Z 
2022-11-23T02:25:07.2193075Z 1112         240 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2193235Z              242 LOAD_ATTR               27 (find_unused_parameters)
2022-11-23T02:25:07.2193372Z              244 POP_JUMP_IF_FALSE      137 (to 274)
2022-11-23T02:25:07.2193499Z              246 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2193646Z              248 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.2193784Z              250 POP_JUMP_IF_TRUE       137 (to 274)
2022-11-23T02:25:07.2193803Z 
2022-11-23T02:25:07.2193930Z 1114         252 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2194046Z              254 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.2194204Z              256 LOAD_METHOD             29 (prepare_for_backward)
2022-11-23T02:25:07.2194223Z 
2022-11-23T02:25:07.2194350Z 1115         258 LOAD_GLOBAL             30 (list)
2022-11-23T02:25:07.2194493Z              260 LOAD_GLOBAL             31 (_find_tensors)
2022-11-23T02:25:07.2194683Z              262 LOAD_FAST                5 (output)
2022-11-23T02:25:07.2194812Z              264 CALL_FUNCTION            1
2022-11-23T02:25:07.2194939Z              266 CALL_FUNCTION            1
2022-11-23T02:25:07.2194958Z 
2022-11-23T02:25:07.2195083Z 1114         268 CALL_METHOD              1
2022-11-23T02:25:07.2195171Z              270 POP_TOP
2022-11-23T02:25:07.2195312Z              272 JUMP_FORWARD            10 (to 294)
2022-11-23T02:25:07.2195331Z 
2022-11-23T02:25:07.2195456Z 1118     >>  274 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2195587Z              276 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.2195743Z              278 LOAD_METHOD             29 (prepare_for_backward)
2022-11-23T02:25:07.2195866Z              280 BUILD_LIST               0
2022-11-23T02:25:07.2195987Z              282 CALL_METHOD              1
2022-11-23T02:25:07.2196073Z              284 POP_TOP
2022-11-23T02:25:07.2196207Z              286 JUMP_FORWARD             3 (to 294)
2022-11-23T02:25:07.2196229Z 
2022-11-23T02:25:07.2196357Z 1120     >>  288 LOAD_CONST               6 (False)
2022-11-23T02:25:07.2196483Z              290 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2196649Z              292 STORE_ATTR              26 (require_forward_param_sync)
2022-11-23T02:25:07.2196757Z          >>  294 POP_BLOCK
2022-11-23T02:25:07.2196776Z 
2022-11-23T02:25:07.2196909Z 1057         296 LOAD_CONST               0 (None)
2022-11-23T02:25:07.2197011Z              298 DUP_TOP
2022-11-23T02:25:07.2197095Z              300 DUP_TOP
2022-11-23T02:25:07.2197222Z              302 CALL_FUNCTION            3
2022-11-23T02:25:07.2197322Z              304 POP_TOP
2022-11-23T02:25:07.2197458Z              306 JUMP_FORWARD             8 (to 324)
2022-11-23T02:25:07.2197575Z          >>  308 WITH_EXCEPT_START
2022-11-23T02:25:07.2197712Z              310 POP_JUMP_IF_TRUE       157 (to 314)
2022-11-23T02:25:07.2197834Z              312 RERAISE                  1
2022-11-23T02:25:07.2197925Z          >>  314 POP_TOP
2022-11-23T02:25:07.2198025Z              316 POP_TOP
2022-11-23T02:25:07.2198124Z              318 POP_TOP
2022-11-23T02:25:07.2198231Z              320 POP_EXCEPT
2022-11-23T02:25:07.2198329Z              322 POP_TOP
2022-11-23T02:25:07.2198348Z 
2022-11-23T02:25:07.2198475Z 1124     >>  324 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2198690Z              326 LOAD_ATTR               27 (find_unused_parameters)
2022-11-23T02:25:07.2198820Z              328 POP_JUMP_IF_FALSE      168 (to 336)
2022-11-23T02:25:07.2198948Z              330 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2199091Z              332 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.2199232Z              334 POP_JUMP_IF_FALSE      178 (to 356)
2022-11-23T02:25:07.2199250Z 
2022-11-23T02:25:07.2199378Z 1125     >>  336 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2199519Z              338 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.2199538Z 
2022-11-23T02:25:07.2199668Z 1124         340 EXTENDED_ARG             1
2022-11-23T02:25:07.2199807Z              342 POP_JUMP_IF_FALSE      265 (to 530)
2022-11-23T02:25:07.2199825Z 
2022-11-23T02:25:07.2199936Z 1125         344 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2200085Z              346 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.2200216Z              348 LOAD_CONST               2 (1)
2022-11-23T02:25:07.2200348Z              350 COMPARE_OP               2 (==)
2022-11-23T02:25:07.2200367Z 
2022-11-23T02:25:07.2200494Z 1124         352 EXTENDED_ARG             1
2022-11-23T02:25:07.2200632Z              354 POP_JUMP_IF_FALSE      265 (to 530)
2022-11-23T02:25:07.2200650Z 
2022-11-23T02:25:07.2200776Z 1128     >>  356 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2200918Z              358 LOAD_ATTR               28 (static_graph)
2022-11-23T02:25:07.2200937Z 
2022-11-23T02:25:07.2201048Z 1129         360 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2201195Z              362 LOAD_ATTR                8 (num_iterations)
2022-11-23T02:25:07.2201266Z 
2022-11-23T02:25:07.2201547Z 1127         364 LOAD_CONST               8 (('static_graph', 'num_iterations'))
2022-11-23T02:25:07.2201680Z              366 BUILD_CONST_KEY_MAP      2
2022-11-23T02:25:07.2201821Z              368 STORE_FAST               6 (state_dict)
2022-11-23T02:25:07.2201841Z 
2022-11-23T02:25:07.2202005Z 1136         370 LOAD_GLOBAL             32 (_tree_flatten_with_rref)
2022-11-23T02:25:07.2202140Z              372 LOAD_FAST                5 (output)
2022-11-23T02:25:07.2202268Z              374 CALL_FUNCTION            1
2022-11-23T02:25:07.2202286Z 
2022-11-23T02:25:07.2202416Z 1132         376 UNPACK_SEQUENCE          3
2022-11-23T02:25:07.2202435Z 
2022-11-23T02:25:07.2202573Z 1133         378 STORE_FAST               7 (output_tensor_list)
2022-11-23T02:25:07.2202592Z 
2022-11-23T02:25:07.2202732Z 1134         380 STORE_FAST               8 (treespec)
2022-11-23T02:25:07.2202751Z 
2022-11-23T02:25:07.2202900Z 1135         382 STORE_FAST               9 (output_is_rref)
2022-11-23T02:25:07.2202919Z 
2022-11-23T02:25:07.2203403Z 1137         384 LOAD_CONST               9 (<code object <listcomp> at 0x7f98dbcd8660, file "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1137>)
2022-11-23T02:25:07.2203731Z              386 LOAD_CONST              10 ('DistributedDataParallel.forward.<locals>.<listcomp>')
2022-11-23T02:25:07.2203860Z              388 MAKE_FUNCTION            0
2022-11-23T02:25:07.2203992Z              390 LOAD_GLOBAL             33 (range)
2022-11-23T02:25:07.2204122Z              392 LOAD_GLOBAL             34 (len)
2022-11-23T02:25:07.2204273Z              394 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.2204385Z              396 CALL_FUNCTION            1
2022-11-23T02:25:07.2204510Z              398 CALL_FUNCTION            1
2022-11-23T02:25:07.2204612Z              400 GET_ITER
2022-11-23T02:25:07.2204736Z              402 CALL_FUNCTION            1
2022-11-23T02:25:07.2204900Z              404 STORE_FAST              10 (output_placeholders)
2022-11-23T02:25:07.2204920Z 
2022-11-23T02:25:07.2205058Z 1140         406 LOAD_GLOBAL             35 (enumerate)
2022-11-23T02:25:07.2205208Z              408 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.2205334Z              410 CALL_FUNCTION            1
2022-11-23T02:25:07.2205474Z              412 GET_ITER
2022-11-23T02:25:07.2205616Z          >>  414 FOR_ITER                18 (to 452)
2022-11-23T02:25:07.2205747Z              416 UNPACK_SEQUENCE          2
2022-11-23T02:25:07.2205878Z              418 STORE_FAST              11 (i)
2022-11-23T02:25:07.2206011Z              420 STORE_FAST               5 (output)
2022-11-23T02:25:07.2206030Z 
2022-11-23T02:25:07.2206163Z 1141         422 LOAD_GLOBAL              0 (torch)
2022-11-23T02:25:07.2206299Z              424 LOAD_METHOD             36 (is_tensor)
2022-11-23T02:25:07.2206415Z              426 LOAD_FAST                5 (output)
2022-11-23T02:25:07.2206542Z              428 CALL_METHOD              1
2022-11-23T02:25:07.2206681Z              430 POP_JUMP_IF_FALSE      225 (to 450)
2022-11-23T02:25:07.2206811Z              432 LOAD_FAST                5 (output)
2022-11-23T02:25:07.2206945Z              434 LOAD_ATTR               37 (grad_fn)
2022-11-23T02:25:07.2207077Z              436 LOAD_CONST               0 (None)
2022-11-23T02:25:07.2207198Z              438 IS_OP                    0
2022-11-23T02:25:07.2207320Z              440 POP_JUMP_IF_FALSE      225 (to 450)
2022-11-23T02:25:07.2207355Z 
2022-11-23T02:25:07.2207466Z 1142         442 LOAD_FAST                5 (output)
2022-11-23T02:25:07.2207623Z              444 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.2207751Z              446 LOAD_FAST               11 (i)
2022-11-23T02:25:07.2207860Z              448 STORE_SUBSCR
2022-11-23T02:25:07.2207994Z          >>  450 JUMP_ABSOLUTE          207 (to 414)
2022-11-23T02:25:07.2208079Z 
2022-11-23T02:25:07.2208224Z 1149     >>  452 LOAD_GLOBAL             38 (_DDPSink)
2022-11-23T02:25:07.2208355Z              454 LOAD_ATTR               39 (apply)
2022-11-23T02:25:07.2208374Z 
2022-11-23T02:25:07.2208486Z 1150         456 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2208621Z              458 LOAD_ATTR                9 (reducer)
2022-11-23T02:25:07.2208640Z 
2022-11-23T02:25:07.2208780Z 1151         460 LOAD_FAST                6 (state_dict)
2022-11-23T02:25:07.2208800Z 
2022-11-23T02:25:07.2208924Z 1149         462 BUILD_LIST               2
2022-11-23T02:25:07.2208943Z 
2022-11-23T02:25:07.2209093Z 1152         464 LOAD_FAST                7 (output_tensor_list)
2022-11-23T02:25:07.2209112Z 
2022-11-23T02:25:07.2209232Z 1149         466 LIST_EXTEND              1
2022-11-23T02:25:07.2209343Z              468 LIST_TO_TUPLE
2022-11-23T02:25:07.2209474Z              470 CALL_FUNCTION_EX         0
2022-11-23T02:25:07.2209641Z              472 STORE_FAST              12 (passthrough_tensor_list)
2022-11-23T02:25:07.2209664Z 
2022-11-23T02:25:07.2209780Z 1154         474 LOAD_GLOBAL             33 (range)
2022-11-23T02:25:07.2209908Z              476 LOAD_GLOBAL             34 (len)
2022-11-23T02:25:07.2210063Z              478 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.2210189Z              480 CALL_FUNCTION            1
2022-11-23T02:25:07.2210317Z              482 CALL_FUNCTION            1
2022-11-23T02:25:07.2210419Z              484 GET_ITER
2022-11-23T02:25:07.2210551Z          >>  486 FOR_ITER                15 (to 518)
2022-11-23T02:25:07.2210663Z              488 STORE_FAST              11 (i)
2022-11-23T02:25:07.2210698Z 
2022-11-23T02:25:07.2210838Z 1155         490 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.2210965Z              492 LOAD_FAST               11 (i)
2022-11-23T02:25:07.2211075Z              494 BINARY_SUBSCR
2022-11-23T02:25:07.2211205Z              496 LOAD_CONST               0 (None)
2022-11-23T02:25:07.2211328Z              498 IS_OP                    0
2022-11-23T02:25:07.2211455Z              500 EXTENDED_ARG             1
2022-11-23T02:25:07.2211595Z              502 POP_JUMP_IF_FALSE      258 (to 516)
2022-11-23T02:25:07.2211614Z 
2022-11-23T02:25:07.2211764Z 1156         504 LOAD_FAST               12 (passthrough_tensor_list)
2022-11-23T02:25:07.2211947Z              506 LOAD_FAST               11 (i)
2022-11-23T02:25:07.2212066Z              508 BINARY_SUBSCR
2022-11-23T02:25:07.2212223Z              510 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.2212351Z              512 LOAD_FAST               11 (i)
2022-11-23T02:25:07.2212460Z              514 STORE_SUBSCR
2022-11-23T02:25:07.2212598Z          >>  516 JUMP_ABSOLUTE          243 (to 486)
2022-11-23T02:25:07.2212617Z 
2022-11-23T02:25:07.2212782Z 1159     >>  518 LOAD_GLOBAL             40 (_tree_unflatten_with_rref)
2022-11-23T02:25:07.2212801Z 
2022-11-23T02:25:07.2212934Z 1160         520 LOAD_FAST               10 (output_placeholders)
2022-11-23T02:25:07.2213076Z              522 LOAD_FAST                8 (treespec)
2022-11-23T02:25:07.2213221Z              524 LOAD_FAST                9 (output_is_rref)
2022-11-23T02:25:07.2213240Z 
2022-11-23T02:25:07.2213369Z 1159         526 CALL_FUNCTION            3
2022-11-23T02:25:07.2213502Z              528 STORE_FAST               5 (output)
2022-11-23T02:25:07.2213524Z 
2022-11-23T02:25:07.2213652Z 1162     >>  530 LOAD_FAST                5 (output)
2022-11-23T02:25:07.2213761Z              532 RETURN_VALUE
2022-11-23T02:25:07.2213852Z  
2022-11-23T02:25:07.2213871Z 
2022-11-23T02:25:07.2214321Z [2022-11-23 02:24:54,467] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1057
2022-11-23T02:25:07.2214624Z [2022-11-23 02:24:54,467] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch []
2022-11-23T02:25:07.2215117Z [2022-11-23 02:24:54,468] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR autograd [TorchVariable(<module 'torch' from '/opt/conda/lib/python3.10/site-packages/torch/__init__.py'>)]
2022-11-23T02:25:07.2215717Z [2022-11-23 02:24:54,468] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR profiler [TorchVariable(<module 'torch.autograd' from '/opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py'>)]
2022-11-23T02:25:07.2216291Z [2022-11-23 02:24:54,468] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR record_function [TorchVariable(<module 'torch.autograd.profiler' from '/opt/conda/lib/python3.10/site-packages/torch/autograd/profiler.py'>)]
2022-11-23T02:25:07.2216730Z [2022-11-23 02:24:54,468] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1058
2022-11-23T02:25:07.2217236Z [2022-11-23 02:24:54,469] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST DistributedDataParallel.forward [TorchVariable(<class 'torch.autograd.profiler.record_function'>)]
2022-11-23T02:25:07.2217674Z [2022-11-23 02:24:54,469] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1057
2022-11-23T02:25:07.2218153Z [2022-11-23 02:24:54,469] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TorchVariable(<class 'torch.autograd.profiler.record_function'>), ConstantVariable(str)]
2022-11-23T02:25:07.2218455Z [2022-11-23 02:24:54,469] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored
2022-11-23T02:25:07.2218794Z [2022-11-23 02:24:54,469] torch._dynamo.symbolic_convert: [DEBUG] TRACE SETUP_WITH 308 [NullContextVariable()]
2022-11-23T02:25:07.2219202Z [2022-11-23 02:24:54,469] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType)]
2022-11-23T02:25:07.2219617Z [2022-11-23 02:24:54,469] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1060
2022-11-23T02:25:07.2219980Z [2022-11-23 02:24:54,469] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL torch [WithExitFunctionVariable()]
2022-11-23T02:25:07.2220543Z [2022-11-23 02:24:54,469] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR is_grad_enabled [WithExitFunctionVariable(), TorchVariable(<module 'torch' from '/opt/conda/lib/python3.10/site-packages/torch/__init__.py'>)]
2022-11-23T02:25:07.2221058Z [2022-11-23 02:24:54,470] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), TorchVariable(<built-in function is_grad_enabled>)]
2022-11-23T02:25:07.2221486Z [2022-11-23 02:24:54,470] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 78 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.2221843Z [2022-11-23 02:24:54,470] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.2222381Z [2022-11-23 02:24:54,470] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR require_backward_grad_sync [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.2222798Z [2022-11-23 02:24:54,470] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 78 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.2223240Z [2022-11-23 02:24:54,470] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1061
2022-11-23T02:25:07.2223597Z [2022-11-23 02:24:54,471] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.2224091Z [2022-11-23 02:24:54,471] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR logger [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.2224523Z [2022-11-23 02:24:54,471] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger)]
2022-11-23T02:25:07.2225044Z [2022-11-23 02:24:54,471] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 1 [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger), ConstantVariable(NoneType)]
2022-11-23T02:25:07.2225458Z [2022-11-23 02:24:54,472] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 44 [WithExitFunctionVariable(), ConstantVariable(bool)]
2022-11-23T02:25:07.2225895Z [2022-11-23 02:24:54,472] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1062
2022-11-23T02:25:07.2226247Z [2022-11-23 02:24:54,472] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [WithExitFunctionVariable()]
2022-11-23T02:25:07.2226741Z [2022-11-23 02:24:54,472] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR logger [WithExitFunctionVariable(), UnspecializedNNModuleVariable(DistributedDataParallel)]
2022-11-23T02:25:07.2227403Z [2022-11-23 02:24:54,472] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR set_runtime_stats_and_log [WithExitFunctionVariable(), UserDefinedObjectVariable(Logger)]
2022-11-23T02:25:07.2227863Z [2022-11-23 02:24:54,473] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), UserDefinedObjectVariable(instancemethod)]
2022-11-23T02:25:07.2228313Z [2022-11-23 02:24:54,474] torch._dynamo.convert_frame: [CODE] ORIGINAL BYTECODE opt_fn /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 415 
2022-11-23T02:25:07.2228448Z 417           0 LOAD_DEREF               0 (ddp_m)
2022-11-23T02:25:07.2228586Z               2 LOAD_FAST                0 (inputs)
2022-11-23T02:25:07.2228697Z               4 CALL_FUNCTION            1
2022-11-23T02:25:07.2228811Z               6 RETURN_VALUE
2022-11-23T02:25:07.2228831Z 
2022-11-23T02:25:07.2228920Z  
2022-11-23T02:25:07.2229369Z [2022-11-23 02:24:54,474] torch._dynamo.convert_frame: [CODE] MODIFIED BYTECODE opt_fn /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 415 
2022-11-23T02:25:07.2229505Z 415           0 LOAD_DEREF               0 (ddp_m)
2022-11-23T02:25:07.2229635Z               2 LOAD_FAST                0 (inputs)
2022-11-23T02:25:07.2229655Z 
2022-11-23T02:25:07.2229782Z 417           4 CALL_FUNCTION            1
2022-11-23T02:25:07.2229892Z               6 RETURN_VALUE
2022-11-23T02:25:07.2229911Z 
2022-11-23T02:25:07.2229983Z  
2022-11-23T02:25:07.2230319Z [2022-11-23 02:24:54,474] torch._dynamo.convert_frame: [CODE] GUARDS:
2022-11-23T02:25:07.2230438Z  - 
2022-11-23T02:25:07.2230613Z             local 'ddp_m' TYPE_MATCH
2022-11-23T02:25:07.2230710Z             {
2022-11-23T02:25:07.2230903Z                 'guard_types': ['TYPE_MATCH'],
2022-11-23T02:25:07.2231134Z                 'code': ['___check_type_id(ddp_m, 94163481642224)'],
2022-11-23T02:25:07.2231475Z                 'obj_weakref': <weakref at 0x7f98ad769c60; to 'DistributedDataParallel' at 0x7f98ad7de680>
2022-11-23T02:25:07.2231823Z                 'guarded_class': <weakref at 0x7f98dbce1a80; to 'ABCMeta' at 0x55a424dfc4f0 (DistributedDataParallel)>
2022-11-23T02:25:07.2231922Z             }
2022-11-23T02:25:07.2232016Z             
2022-11-23T02:25:07.2232118Z  - 
2022-11-23T02:25:07.2232293Z             local 'inputs' TENSOR_MATCH
2022-11-23T02:25:07.2232386Z             {
2022-11-23T02:25:07.2232580Z                 'guard_types': ['TENSOR_MATCH'],
2022-11-23T02:25:07.2232713Z                 'code': None,
2022-11-23T02:25:07.2232999Z                 'obj_weakref': <weakref at 0x7f98ad628400; to 'Tensor' at 0x7f98ad8f9530>
2022-11-23T02:25:07.2233330Z                 'guarded_class': <weakref at 0x7f98dc435d50; to 'torch._C._TensorMeta' at 0x55a4248e47a0 (Tensor)>
2022-11-23T02:25:07.2233423Z             }
2022-11-23T02:25:07.2233516Z             
2022-11-23T02:25:07.2233845Z [2022-11-23 02:24:54,476] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
2022-11-23T02:25:07.2234273Z [2022-11-23 02:24:54,476] torch._dynamo.symbolic_convert: [DEBUG] TRACE starts_line /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py:54
2022-11-23T02:25:07.2234646Z [2022-11-23 02:24:54,476] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
2022-11-23T02:25:07.2234955Z [2022-11-23 02:24:54,476] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR net [NNModuleVariable()]
2022-11-23T02:25:07.2235293Z [2022-11-23 02:24:54,477] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST inputs [NNModuleVariable()]
2022-11-23T02:25:07.2235660Z [2022-11-23 02:24:54,477] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
2022-11-23T02:25:07.2235992Z [2022-11-23 02:24:54,513] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [TensorVariable()]
2022-11-23T02:25:07.2236310Z [2022-11-23 02:24:54,513] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward
2022-11-23T02:25:07.2236627Z [2022-11-23 02:24:54,514] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
2022-11-23T02:25:07.2237057Z [2022-11-23 02:24:54,515] torch._dynamo.optimizations.distributed: [INFO] DDPOptimizer used bucket cap 262144000 and produced the following buckets:
2022-11-23T02:25:07.2237482Z [2022-11-23 02:24:54,515] torch._dynamo.optimizations.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
2022-11-23T02:25:07.2237801Z [2022-11-23 02:24:54,515] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
2022-11-23T02:25:07.2238040Z [2022-11-23 02:24:54,515] torch._dynamo.output_graph: [CODE] TRACED GRAPH
2022-11-23T02:25:07.2238225Z  __compiled_fn_5 <eval_with_key>.61 opcode, name, target, args, kwargs
2022-11-23T02:25:07.2238363Z placeholder, inputs, inputs, (), {}
2022-11-23T02:25:07.2238515Z call_module, self_net_0, self_net_0, (inputs,), {}
2022-11-23T02:25:07.2238672Z call_module, self_net_1, self_net_1, (self_net_0,), {}
2022-11-23T02:25:07.2238828Z call_module, self_net_2, self_net_2, (self_net_1,), {}
2022-11-23T02:25:07.2238979Z call_module, self_net_3, self_net_3, (self_net_2,), {}
2022-11-23T02:25:07.2239116Z call_module, self_net_4, self_net_4, (self_net_3,), {}
2022-11-23T02:25:07.2239267Z call_module, self_net_5, self_net_5, (self_net_4,), {}
2022-11-23T02:25:07.2239418Z call_module, self_net_6, self_net_6, (self_net_5,), {}
2022-11-23T02:25:07.2239569Z call_module, self_net_7, self_net_7, (self_net_6,), {}
2022-11-23T02:25:07.2239770Z output, output, output, ((self_net_7,),), {}
2022-11-23T02:25:07.2239792Z 
2022-11-23T02:25:07.2240261Z [2022-11-23 02:24:54,516] torch._dynamo.convert_frame: [CODE] ORIGINAL BYTECODE forward /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 53 
2022-11-23T02:25:07.2240395Z  54           0 LOAD_FAST                0 (self)
2022-11-23T02:25:07.2240527Z               2 LOAD_METHOD              0 (net)
2022-11-23T02:25:07.2240660Z               4 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.2240767Z               6 CALL_METHOD              1
2022-11-23T02:25:07.2240881Z               8 RETURN_VALUE
2022-11-23T02:25:07.2240900Z 
2022-11-23T02:25:07.2240988Z  
2022-11-23T02:25:07.2241444Z [2022-11-23 02:24:54,516] torch._dynamo.convert_frame: [CODE] MODIFIED BYTECODE forward /var/lib/jenkins/workspace/test/distributed/test_dynamo_distributed.py line 53 
2022-11-23T02:25:07.2241595Z  53           0 LOAD_GLOBAL              1 (__compiled_fn_5)
2022-11-23T02:25:07.2241731Z               2 LOAD_FAST                1 (inputs)
2022-11-23T02:25:07.2241858Z               4 CALL_FUNCTION            1
2022-11-23T02:25:07.2241971Z               6 UNPACK_SEQUENCE          1
2022-11-23T02:25:07.2242080Z               8 RETURN_VALUE
2022-11-23T02:25:07.2242100Z 
2022-11-23T02:25:07.2242189Z  
2022-11-23T02:25:07.2242440Z [2022-11-23 02:24:54,516] torch._dynamo.convert_frame: [CODE] GUARDS:
2022-11-23T02:25:07.2242541Z  - 
2022-11-23T02:25:07.2242705Z             local 'self' NN_MODULE
2022-11-23T02:25:07.2242800Z             {
2022-11-23T02:25:07.2242968Z                 'guard_types': ['ID_MATCH'],
2022-11-23T02:25:07.2243262Z                 'code': ['___check_obj_id(self, 140293722193680)'],
2022-11-23T02:25:07.2243554Z                 'obj_weakref': <weakref at 0x7f98ad72ad40; to 'ToyModel' at 0x7f98ad79bf10>
2022-11-23T02:25:07.2243864Z                 'guarded_class': <weakref at 0x7f9a4b8d0630; to 'type' at 0x55a4264bb360 (ToyModel)>
2022-11-23T02:25:07.2243959Z             }
2022-11-23T02:25:07.2244057Z             
2022-11-23T02:25:07.2244158Z  - 
2022-11-23T02:25:07.2244317Z             local 'inputs' TENSOR_MATCH
2022-11-23T02:25:07.2244410Z             {
2022-11-23T02:25:07.2244603Z                 'guard_types': ['TENSOR_MATCH'],
2022-11-23T02:25:07.2244752Z                 'code': None,
2022-11-23T02:25:07.2245036Z                 'obj_weakref': <weakref at 0x7f98ad628400; to 'Tensor' at 0x7f98ad8f9530>
2022-11-23T02:25:07.2245369Z                 'guarded_class': <weakref at 0x7f98dc435d50; to 'torch._C._TensorMeta' at 0x55a4248e47a0 (Tensor)>
2022-11-23T02:25:07.2245464Z             }
2022-11-23T02:25:07.2245562Z             
2022-11-23T02:25:07.2245649Z  - 
2022-11-23T02:25:07.2245846Z             local_nn_module 'self.net' NN_MODULE
2022-11-23T02:25:07.2245938Z             {
2022-11-23T02:25:07.2246102Z                 'guard_types': None,
2022-11-23T02:25:07.2246248Z                 'code': None,
2022-11-23T02:25:07.2246410Z                 'obj_weakref': None
2022-11-23T02:25:07.2246563Z                 'guarded_class': None
2022-11-23T02:25:07.2246655Z             }
2022-11-23T02:25:07.2246747Z             
2022-11-23T02:25:07.2246847Z  - 
2022-11-23T02:25:07.2247051Z             local_nn_module 'self.net[0]' NN_MODULE
2022-11-23T02:25:07.2247145Z             {
2022-11-23T02:25:07.2247310Z                 'guard_types': None,
2022-11-23T02:25:07.2247442Z                 'code': None,
2022-11-23T02:25:07.2247601Z                 'obj_weakref': None
2022-11-23T02:25:07.2247765Z                 'guarded_class': None
2022-11-23T02:25:07.2247857Z             }
2022-11-23T02:25:07.2247954Z             
2022-11-23T02:25:07.2248054Z  - 
2022-11-23T02:25:07.2248255Z             local_nn_module 'self.net[1]' NN_MODULE
2022-11-23T02:25:07.2248332Z             {
2022-11-23T02:25:07.2248493Z                 'guard_types': None,
2022-11-23T02:25:07.2248686Z                 'code': None,
2022-11-23T02:25:07.2248897Z                 'obj_weakref': None
2022-11-23T02:25:07.2249170Z                 'guarded_class': None
2022-11-23T02:25:07.2249311Z             }
2022-11-23T02:25:07.2249387Z             
2022-11-23T02:25:07.2249647Z  - 
2022-11-23T02:25:07.2249952Z             local_nn_module 'self.net[2]' NN_MODULE
2022-11-23T02:25:07.2250083Z             {
2022-11-23T02:25:07.2250234Z                 'guard_types': None,
2022-11-23T02:25:07.2250420Z                 'code': None,
2022-11-23T02:25:07.2250620Z                 'obj_weakref': None
2022-11-23T02:25:07.2250835Z                 'guarded_class': None
2022-11-23T02:25:07.2250966Z             }
2022-11-23T02:25:07.2251097Z             
2022-11-23T02:25:07.2251284Z  - 
2022-11-23T02:25:07.2251481Z             local_nn_module 'self.net[3]' NN_MODULE
2022-11-23T02:25:07.2251610Z             {
2022-11-23T02:25:07.2251811Z                 'guard_types': None,
2022-11-23T02:25:07.2252000Z                 'code': None,
2022-11-23T02:25:07.2252211Z                 'obj_weakref': None
2022-11-23T02:25:07.2252416Z                 'guarded_class': None
2022-11-23T02:25:07.2252552Z             }
2022-11-23T02:25:07.2252630Z             
2022-11-23T02:25:07.2252771Z  - 
2022-11-23T02:25:07.2253149Z             local_nn_module 'self.net[4]' NN_MODULE
2022-11-23T02:25:07.2253285Z             {
2022-11-23T02:25:07.2253495Z                 'guard_types': None,
2022-11-23T02:25:07.2253693Z                 'code': None,
2022-11-23T02:25:07.2253901Z                 'obj_weakref': None
2022-11-23T02:25:07.2254053Z                 'guarded_class': None
2022-11-23T02:25:07.2254184Z             }
2022-11-23T02:25:07.2254315Z             
2022-11-23T02:25:07.2254451Z  - 
2022-11-23T02:25:07.2254803Z             local_nn_module 'self.net[5]' NN_MODULE
2022-11-23T02:25:07.2254937Z             {
2022-11-23T02:25:07.2255090Z                 'guard_types': None,
2022-11-23T02:25:07.2255287Z                 'code': None,
2022-11-23T02:25:07.2255489Z                 'obj_weakref': None
2022-11-23T02:25:07.2255693Z                 'guarded_class': None
2022-11-23T02:25:07.2255828Z             }
2022-11-23T02:25:07.2255962Z             
2022-11-23T02:25:07.2260396Z  - 
2022-11-23T02:25:07.2260675Z             local_nn_module 'self.net[6]' NN_MODULE
2022-11-23T02:25:07.2260898Z             {
2022-11-23T02:25:07.2261131Z                 'guard_types': None,
2022-11-23T02:25:07.2261323Z                 'code': None,
2022-11-23T02:25:07.2261530Z                 'obj_weakref': None
2022-11-23T02:25:07.2261734Z                 'guarded_class': None
2022-11-23T02:25:07.2261865Z             }
2022-11-23T02:25:07.2261944Z             
2022-11-23T02:25:07.2262084Z  - 
2022-11-23T02:25:07.2262339Z             local_nn_module 'self.net[7]' NN_MODULE
2022-11-23T02:25:07.2262524Z             {
2022-11-23T02:25:07.2262733Z                 'guard_types': None,
2022-11-23T02:25:07.2262923Z                 'code': None,
2022-11-23T02:25:07.2263073Z                 'obj_weakref': None
2022-11-23T02:25:07.2263281Z                 'guarded_class': None
2022-11-23T02:25:07.2263411Z             }
2022-11-23T02:25:07.2263547Z             
2022-11-23T02:25:07.2263756Z frames [('total', 2), ('ok', 2)]
2022-11-23T02:25:07.2264165Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2022-11-23T02:25:07.2264350Z unimplemented []
2022-11-23T02:25:07.2264649Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2022-11-23T02:25:07.2264963Z stats [('calls_captured', 8), ('fusions_possible', 7), ('unique_graphs', 1)]
2022-11-23T02:25:07.2282826Z ok (0.059s)
2022-11-23T02:25:07.2283269Z   test_ddp_baseline_aot_eager_multiprocess (__main__.TestDistributedMultiProc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58122
2022-11-23T02:25:07.2283524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58123
2022-11-23T02:25:07.2283961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.2284314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.2284751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.2284952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.2285341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.2285528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.2285912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.2286116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.2286359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:07.2286602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:07.2286861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:07.2287118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:07.2287547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.2287967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.2288071Z ok (5.715s)
2022-11-23T02:25:07.2288481Z   test_fsdp_aot_eager (__main__.TestDistributedMultiProc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58210
2022-11-23T02:25:07.2288715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58211
2022-11-23T02:25:07.2289111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.2289297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.2289698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.2289901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.2290291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:07.2290474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:07.2290879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:07.2291066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:07.2291309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:07.2291552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:07.2291808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:07.2292065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:07.2292487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.2292905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:07.2293013Z ok (6.416s)
2022-11-23T02:25:07.2293285Z   test_fsdp_inductor (__main__.TestDistributedMultiProc) ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s)
2022-11-23T02:25:07.2293573Z   test_hf_bert_ddp (__main__.TestDistributedMultiProc) ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s)
2022-11-23T02:25:07.2293920Z   test_hf_bert_fsdp (__main__.TestDistributedMultiProc) ... skip: Inductor+gpu needs triton and recent GPU arch (0.002s)
2022-11-23T02:25:07.2293945Z 
2022-11-23T02:25:07.2294245Z ----------------------------------------------------------------------
2022-11-23T02:25:07.2294365Z Ran 14 tests in 16.488s
2022-11-23T02:25:07.2294386Z 
2022-11-23T02:25:07.2294496Z OK (skipped=6)
2022-11-23T02:25:07.2294515Z 
2022-11-23T02:25:07.2294643Z Generating XML reports...
2022-11-23T02:25:07.2295101Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributed-20221123022450.xml
2022-11-23T02:25:07.2295593Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributedMultiProc-20221123022450.xml
2022-11-23T02:25:07.2295619Z 
2022-11-23T02:25:07.2296009Z ##[endgroup]
2022-11-23T02:25:07.2296510Z FINISHED PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/workspace/test/test-reports/distributed-test_dynamo_distributed_gmpwz2lt)
2022-11-23T02:25:07.2296530Z 
2022-11-23T02:25:07.2296832Z Running distributed/_composable/test_fully_shard ... [2022-11-23 02:25:07.149633]
2022-11-23T02:25:07.2297350Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_fully_shard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:25:07.149893]
2022-11-23T02:25:31.4326613Z 
2022-11-23T02:25:31.4327072Z Expand the folded group to see the log file of distributed/_composable/test_fully_shard
2022-11-23T02:25:31.4332209Z ##[group]PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_fully_shard_ezk370m8)
2022-11-23T02:25:31.4333007Z 
2022-11-23T02:25:31.4333109Z Running tests...
2022-11-23T02:25:31.4333638Z ----------------------------------------------------------------------
2022-11-23T02:25:31.4334677Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_fully_shard
2022-11-23T02:25:31.4335163Z   test_auto_wrap_policy (__main__.TestFSDPInitialization)
2022-11-23T02:25:31.4335808Z Tests passing an ``auto_wrap_policy``. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:31.4336493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58333
2022-11-23T02:25:31.4336954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58334
2022-11-23T02:25:31.4337604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:31.4338065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:31.4338644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:31.4339098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:31.4339654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:31.4340134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:31.4340744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:31.4341214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:31.4341655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:31.4342160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:31.4342824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:31.4343532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:31.4344035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:31.4344654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:31.4345044Z dist init r=0, world=2
2022-11-23T02:25:31.4345285Z dist init r=1, world=2
2022-11-23T02:25:31.4345545Z ok (5.534s)
2022-11-23T02:25:31.4345842Z   test_device_id (__main__.TestFSDPInitialization)
2022-11-23T02:25:31.4346304Z Tests passing a ``device_id``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58412
2022-11-23T02:25:31.4347227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58413
2022-11-23T02:25:31.4347978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:31.4348440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:31.4349002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:31.4349486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:31.4350065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:31.4350513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:31.4351068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:31.4351532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:31.4351995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:31.4352668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:31.4358152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:31.4359607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:31.4360745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:31.4361700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:31.4362462Z dist init r=0, world=2
2022-11-23T02:25:31.4362943Z dist init r=1, world=2
2022-11-23T02:25:31.4363455Z ok (4.011s)
2022-11-23T02:25:31.4364080Z   test_materialize_meta_module (__main__.TestFSDPInitialization)
2022-11-23T02:25:31.4364716Z Tests materializing a meta-device module. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58491
2022-11-23T02:25:31.4365242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58492
2022-11-23T02:25:31.4365859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:31.4366295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:31.4366879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:31.4367351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:31.4367931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:31.4368357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:31.4368927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:31.4369398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:31.4369853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:31.4370329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:31.4371168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:31.4371896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:31.4372402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:31.4372878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:31.4373231Z dist init r=1, world=2
2022-11-23T02:25:31.4373489Z dist init r=0, world=2
2022-11-23T02:25:31.4373711Z ok (3.911s)
2022-11-23T02:25:31.4374025Z   test_sync_module_states (__main__.TestFSDPInitialization)
2022-11-23T02:25:31.4374514Z Tests passing ``sync_module_states=True``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58570
2022-11-23T02:25:31.4375001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58571
2022-11-23T02:25:31.4375617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:31.4376065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:31.4376650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:31.4377148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:31.4377732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:31.4378268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:31.4378848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:31.4379297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:31.4379759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:31.4380255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:31.4380897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:31.4381589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:31.4382111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:31.4382589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:31.4382930Z dist init r=1, world=2
2022-11-23T02:25:31.4383180Z dist init r=0, world=2
2022-11-23T02:25:31.4383421Z ok (3.911s)
2022-11-23T02:25:31.4383680Z   test_training (__main__.TestFSDPRuntime)
2022-11-23T02:25:31.4384150Z Tests training (forward, backward, optimizer). ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58649
2022-11-23T02:25:31.4384668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58650
2022-11-23T02:25:31.4385282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:31.4385717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:31.4386282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:31.4386734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:31.4388006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:31.4388487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:31.4389197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:31.4389687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:31.4390124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:31.4390620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:31.4391281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:31.4391974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:31.4392479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:31.4392953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:31.4393314Z dist init r=0, world=2
2022-11-23T02:25:31.4393547Z dist init r=1, world=2
2022-11-23T02:25:31.4393784Z ok (4.613s)
2022-11-23T02:25:31.4393935Z 
2022-11-23T02:25:31.4394210Z ----------------------------------------------------------------------
2022-11-23T02:25:31.4394521Z Ran 5 tests in 21.980s
2022-11-23T02:25:31.4394686Z 
2022-11-23T02:25:31.4394779Z OK
2022-11-23T02:25:31.4394915Z 
2022-11-23T02:25:31.4395042Z Generating XML reports...
2022-11-23T02:25:31.4395674Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPInitialization-20221123022509.xml
2022-11-23T02:25:31.4396552Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPRuntime-20221123022509.xml
2022-11-23T02:25:31.4396902Z 
2022-11-23T02:25:31.4397353Z ##[endgroup]
2022-11-23T02:25:31.4397989Z FINISHED PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_fully_shard_ezk370m8)
2022-11-23T02:25:31.4398357Z 
2022-11-23T02:25:31.4398650Z Running distributed/fsdp/test_fsdp_ignored_modules ... [2022-11-23 02:25:31.432692]
2022-11-23T02:25:31.4399346Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:25:31.433020]
2022-11-23T02:25:57.1790093Z 
2022-11-23T02:25:57.1792634Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_ignored_modules
2022-11-23T02:25:57.1794016Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_q0y69zpz)
2022-11-23T02:25:57.1794416Z 
2022-11-23T02:25:57.1794524Z Running tests...
2022-11-23T02:25:57.1799568Z ----------------------------------------------------------------------
2022-11-23T02:25:57.1800206Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules
2022-11-23T02:25:57.1800769Z   test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_False (__main__.TestFSDPIgnoredModules)
2022-11-23T02:25:57.1801279Z Tests ignoring different modules across ranks. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:25:57.1801755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58767
2022-11-23T02:25:57.1802191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58768
2022-11-23T02:25:57.1804874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:57.1805361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:57.1805975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:57.1806459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:57.1807255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:57.1807735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:57.1808325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:57.1808798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:57.1809245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:57.1809751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:57.1810423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:57.1811114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:57.1811629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:57.1812109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:57.1812468Z dist init r=1, world=2
2022-11-23T02:25:57.1812704Z dist init r=0, world=2
2022-11-23T02:25:57.1812947Z ok (6.060s)
2022-11-23T02:25:57.1813379Z   test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_True (__main__.TestFSDPIgnoredModules)
2022-11-23T02:25:57.1813947Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58850
2022-11-23T02:25:57.1814595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58851
2022-11-23T02:25:57.1815194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:57.1815648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:57.1816230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:57.1816687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:57.1817270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:57.1817716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:57.1818287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:57.1818738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:57.1819199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:57.1819696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:57.1820357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:57.1821029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:57.1821553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:57.1822026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:57.1822368Z dist init r=1, world=2
2022-11-23T02:25:57.1822622Z dist init r=0, world=2
2022-11-23T02:25:57.1822869Z ok (4.412s)
2022-11-23T02:25:57.1823175Z   test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules)
2022-11-23T02:25:57.1823702Z Tests that passing an FSDP module as an ignored module or the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58933
2022-11-23T02:25:57.1824231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58934
2022-11-23T02:25:57.1824917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:57.1825360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:57.1825939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:57.1826411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:57.1827288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:57.1827732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:57.1828311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:57.1828776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:57.1829213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:57.1829709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:57.1830370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:57.1831064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:57.1831567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:57.1832146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:57.1832506Z dist init r=1, world=2
2022-11-23T02:25:57.1832741Z dist init r=0, world=2
2022-11-23T02:25:57.1832986Z ok (3.911s)
2022-11-23T02:25:57.1833311Z   test_ignored_modules_nested (__main__.TestFSDPIgnoredModules)
2022-11-23T02:25:57.1833840Z Tests that passing a module with nested FSDP modules does not ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59012
2022-11-23T02:25:57.1834351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59013
2022-11-23T02:25:57.1834970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:57.1835419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:57.1835979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:57.1836454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:57.1837038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:57.1837485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:57.1838045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:57.1838507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:57.1838960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:57.1839435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:57.1840093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:57.1840786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:57.1841311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:57.1841762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:57.1842191Z dist init r=1, world=2
2022-11-23T02:25:57.1842457Z dist init r=0, world=2
2022-11-23T02:25:57.1842678Z ok (4.313s)
2022-11-23T02:25:57.1843015Z   test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules)
2022-11-23T02:25:57.1843682Z Tests that ignored modules' parameters are not flattened for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59095
2022-11-23T02:25:57.1844219Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59096
2022-11-23T02:25:57.1844809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:57.1845265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:57.1845843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:57.1846314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:57.1846883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:25:57.1847327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:25:57.1847899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:25:57.1848344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:25:57.1848802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:25:57.1849370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:25:57.1850030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:57.1850703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:25:57.1851227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:25:57.1851702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:25:57.1852084Z dist init r=1, world=2
2022-11-23T02:25:57.1852339Z dist init r=0, world=2
2022-11-23T02:25:57.1852583Z ok (4.715s)
2022-11-23T02:25:57.1852734Z 
2022-11-23T02:25:57.1853009Z ----------------------------------------------------------------------
2022-11-23T02:25:57.1853322Z Ran 5 tests in 23.411s
2022-11-23T02:25:57.1853485Z 
2022-11-23T02:25:57.1853589Z OK
2022-11-23T02:25:57.1853724Z 
2022-11-23T02:25:57.1853851Z Generating XML reports...
2022-11-23T02:25:57.1854476Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20221123022533.xml
2022-11-23T02:25:57.1854859Z 
2022-11-23T02:25:57.1855309Z ##[endgroup]
2022-11-23T02:25:57.1855954Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_q0y69zpz)
2022-11-23T02:25:57.1856334Z 
2022-11-23T02:25:57.1856631Z Running distributed/elastic/multiprocessing/api_test ... [2022-11-23 02:25:57.179043]
2022-11-23T02:25:57.1857338Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:25:57.179314]
2022-11-23T02:26:26.9948156Z 
2022-11-23T02:26:26.9948653Z Expand the folded group to see the log file of distributed/elastic/multiprocessing/api_test
2022-11-23T02:26:26.9951542Z ##[group]PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_dnv4i9_j)
2022-11-23T02:26:26.9952011Z 
2022-11-23T02:26:26.9952108Z Running tests...
2022-11-23T02:26:26.9953537Z ----------------------------------------------------------------------
2022-11-23T02:26:26.9954798Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test
2022-11-23T02:26:26.9955664Z   test_get_failures (__main__.RunProcResultsTest) ... ok (1.658s)
2022-11-23T02:26:26.9956240Z   test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s)
2022-11-23T02:26:26.9956651Z   test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.002s)
2022-11-23T02:26:26.9957060Z   test_binary (__main__.StartProcessesListTest) ... hello stderr from 0
2022-11-23T02:26:26.9957392Z hello stdout from 0
2022-11-23T02:26:26.9957660Z hello stderr from 1
2022-11-23T02:26:26.9957918Z hello stdout from 1
2022-11-23T02:26:26.9958144Z ok (0.104s)
2022-11-23T02:26:26.9958636Z   test_binary_exit (__main__.StartProcessesListTest) ... bar stderr from 1
2022-11-23T02:26:26.9959220Z bar stdout from 1
2022-11-23T02:26:26.9959658Z failed (exitcode: 138) local_rank: 0 (pid: 59214) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2022-11-23T02:26:26.9962561Z ok (0.105s)
2022-11-23T02:26:26.9962964Z   test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.002s)
2022-11-23T02:26:26.9963468Z   test_binary_raises (__main__.StartProcessesListTest) ... Traceback (most recent call last):
2022-11-23T02:26:26.9963984Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-11-23T02:26:26.9964385Z     raise RuntimeError(f"raised from {rank}")
2022-11-23T02:26:26.9964696Z RuntimeError: raised from 0
2022-11-23T02:26:26.9964958Z bar from 1
2022-11-23T02:26:26.9965592Z failed (exitcode: 1) local_rank: 0 (pid: 59217) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2022-11-23T02:26:26.9965996Z ok (0.104s)
2022-11-23T02:26:26.9966356Z   test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1
2022-11-23T02:26:26.9966778Z [trainer1]:world stderr from 1
2022-11-23T02:26:26.9967069Z [trainer0]:hello stdout from 0
2022-11-23T02:26:26.9967318Z ok (1.107s)
2022-11-23T02:26:26.9968020Z   test_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:26.9968550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:26.9969124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:26.9969598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:26.9970191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:26.9970653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:26.9971209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:26.9971679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:26.9972013Z hello stderr from 1
2022-11-23T02:26:26.9972250Z hello stderr from 0
2022-11-23T02:26:26.9972502Z hello stdout from 1
2022-11-23T02:26:26.9972750Z hello stdout from 0
2022-11-23T02:26:26.9973008Z Closing process 59224 via signal SIGTERM
2022-11-23T02:26:26.9973286Z ok (2.419s)
2022-11-23T02:26:26.9973933Z   test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:26.9974480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:26.9975034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:26.9975480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:26.9976132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:26.9976605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:26.9977198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:26.9977669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:26.9978250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:26.9978678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:26.9979256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:26.9979723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:26.9980308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:26.9980730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:26.9981302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:26.9981764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:26.9982095Z Closing process 59290 via signal SIGTERM
2022-11-23T02:26:26.9982405Z Closing process 59291 via signal SIGTERM
2022-11-23T02:26:26.9982709Z Closing process 59293 via signal SIGTERM
2022-11-23T02:26:26.9982958Z ok (2.774s)
2022-11-23T02:26:26.9983348Z   test_function_raise (__main__.StartProcessesListTest)
2022-11-23T02:26:26.9984050Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:26.9984567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:26.9985130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:26.9985603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:26.9986186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:26.9986614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:26.9987628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:26.9988108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:26.9988523Z failed (exitcode: 1) local_rank: 0 (pid: 59422) of fn: echo2 (start_method: spawn)
2022-11-23T02:26:26.9988860Z Traceback (most recent call last):
2022-11-23T02:26:26.9989410Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-11-23T02:26:26.9989844Z     self._pc.join(-1)
2022-11-23T02:26:26.9990301Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-11-23T02:26:26.9990766Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-11-23T02:26:26.9991226Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-11-23T02:26:26.9991497Z 
2022-11-23T02:26:26.9991729Z -- Process 0 terminated with the following error:
2022-11-23T02:26:26.9992040Z Traceback (most recent call last):
2022-11-23T02:26:26.9992535Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-11-23T02:26:26.9992905Z     fn(i, *args)
2022-11-23T02:26:26.9993389Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2022-11-23T02:26:26.9993788Z     ret = record(fn)(*args_)
2022-11-23T02:26:26.9994443Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2022-11-23T02:26:26.9994876Z     return f(*args, **kwargs)
2022-11-23T02:26:26.9995253Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-11-23T02:26:26.9995639Z     raise RuntimeError(msg)
2022-11-23T02:26:26.9995911Z RuntimeError: hello
2022-11-23T02:26:26.9996053Z 
2022-11-23T02:26:26.9996154Z ok (2.422s)
2022-11-23T02:26:26.9996487Z   test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s)
2022-11-23T02:26:26.9996906Z   test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.002s)
2022-11-23T02:26:26.9997459Z   test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 59488 via signal SIGTERM
2022-11-23T02:26:26.9997834Z ok (0.003s)
2022-11-23T02:26:26.9998442Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-11-23T02:26:26.9998916Z Traceback (most recent call last):
2022-11-23T02:26:26.9999457Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-11-23T02:26:26.9999885Z     self._pc.join(-1)
2022-11-23T02:26:27.0000199Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2022-11-23T02:26:27.0000551Z     return self._mock_call(*args, **kwargs)
2022-11-23T02:26:27.0000906Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2022-11-23T02:26:27.0001253Z     return self._execute_mock_call(*args, **kwargs)
2022-11-23T02:26:27.0001737Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2022-11-23T02:26:27.0002069Z     raise effect
2022-11-23T02:26:27.0002425Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-11-23T02:26:27.0002785Z ok (0.004s)
2022-11-23T02:26:27.0003424Z   test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:27.0003952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:27.0004510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:27.0004979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:27.0005293Z ok (3.284s)
2022-11-23T02:26:27.0005662Z   test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 59522 closing signal SIGTERM
2022-11-23T02:26:27.0006046Z ok (0.004s)
2022-11-23T02:26:27.0006354Z   test_to_map (__main__.StartProcessesListTest) ... ok (0.002s)
2022-11-23T02:26:27.0006740Z   test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s)
2022-11-23T02:26:27.0007465Z   test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:27.0007992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:27.0008569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:27.0009023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:27.0009600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:27.0010043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:27.0010623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:27.0011069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:27.0011372Z hello
2022-11-23T02:26:27.0011582Z world
2022-11-23T02:26:27.0011882Z Closing process 59524 via signal SIGTERM
2022-11-23T02:26:27.0012168Z ok (2.407s)
2022-11-23T02:26:27.0012493Z   test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s)
2022-11-23T02:26:27.0012880Z   test_binary_exit (__main__.StartProcessesTest) ... bar stderr from 1
2022-11-23T02:26:27.0013200Z bar stdout from 1
2022-11-23T02:26:27.0013631Z failed (exitcode: 138) local_rank: 0 (pid: 59589) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2022-11-23T02:26:27.0014011Z ok (0.104s)
2022-11-23T02:26:27.0014351Z   test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.002s)
2022-11-23T02:26:27.0014765Z   test_binary_raises (__main__.StartProcessesTest) ... bar from 1
2022-11-23T02:26:27.0015103Z Traceback (most recent call last):
2022-11-23T02:26:27.0015505Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2022-11-23T02:26:27.0015919Z     raise RuntimeError(f"raised from {rank}")
2022-11-23T02:26:27.0016221Z RuntimeError: raised from 0
2022-11-23T02:26:27.0016647Z failed (exitcode: 1) local_rank: 0 (pid: 59592) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2022-11-23T02:26:27.0017039Z ok (0.104s)
2022-11-23T02:26:27.0017680Z   test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:27.0018201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:27.0018756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:27.0019305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:27.0019894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:27.0020328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:27.0020906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:27.0021372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:27.0021948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:27.0022374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:27.0022945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:27.0023415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:27.0023998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:27.0024423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:27.0024998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:27.0025463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:27.0025793Z Closing process 59594 via signal SIGTERM
2022-11-23T02:26:27.0026102Z Closing process 59596 via signal SIGTERM
2022-11-23T02:26:27.0026410Z Closing process 59597 via signal SIGTERM
2022-11-23T02:26:27.0026659Z ok (2.784s)
2022-11-23T02:26:27.0027324Z   test_function_raise (__main__.StartProcessesTest)
2022-11-23T02:26:27.0028015Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:27.0028536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:27.0029096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:27.0029655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:27.0030258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:27.0030687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:27.0031261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:27.0031720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:27.0032129Z failed (exitcode: 1) local_rank: 0 (pid: 59726) of fn: echo2 (start_method: spawn)
2022-11-23T02:26:27.0032470Z Traceback (most recent call last):
2022-11-23T02:26:27.0033010Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-11-23T02:26:27.0033437Z     self._pc.join(-1)
2022-11-23T02:26:27.0033896Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2022-11-23T02:26:27.0034362Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2022-11-23T02:26:27.0034819Z torch.multiprocessing.spawn.ProcessRaisedException: 
2022-11-23T02:26:27.0035089Z 
2022-11-23T02:26:27.0035317Z -- Process 0 terminated with the following error:
2022-11-23T02:26:27.0035622Z Traceback (most recent call last):
2022-11-23T02:26:27.0036118Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2022-11-23T02:26:27.0036478Z     fn(i, *args)
2022-11-23T02:26:27.0036965Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2022-11-23T02:26:27.0037456Z     ret = record(fn)(*args_)
2022-11-23T02:26:27.0038009Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2022-11-23T02:26:27.0038419Z     return f(*args, **kwargs)
2022-11-23T02:26:27.0038801Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2022-11-23T02:26:27.0039186Z     raise RuntimeError(msg)
2022-11-23T02:26:27.0039452Z RuntimeError: hello
2022-11-23T02:26:27.0039595Z 
2022-11-23T02:26:27.0039698Z ok (2.322s)
2022-11-23T02:26:27.0040018Z   test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s)
2022-11-23T02:26:27.0040421Z   test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.001s)
2022-11-23T02:26:27.0040861Z   test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 59792 via signal SIGTERM
2022-11-23T02:26:27.0041226Z ok (0.003s)
2022-11-23T02:26:27.0041832Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2022-11-23T02:26:27.0042300Z Traceback (most recent call last):
2022-11-23T02:26:27.0042823Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2022-11-23T02:26:27.0043253Z     self._pc.join(-1)
2022-11-23T02:26:27.0043585Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2022-11-23T02:26:27.0043917Z     return self._mock_call(*args, **kwargs)
2022-11-23T02:26:27.0044274Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2022-11-23T02:26:27.0044637Z     return self._execute_mock_call(*args, **kwargs)
2022-11-23T02:26:27.0044999Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2022-11-23T02:26:27.0045327Z     raise effect
2022-11-23T02:26:27.0045707Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2022-11-23T02:26:27.0046066Z ok (0.004s)
2022-11-23T02:26:27.0046663Z   test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:27.0047244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:27.0047842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:27.0048299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:27.0048611Z ok (3.272s)
2022-11-23T02:26:27.0048992Z   test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 59826 closing signal SIGTERM
2022-11-23T02:26:27.0049354Z ok (0.004s)
2022-11-23T02:26:27.0049639Z   test_to_map (__main__.StartProcessesTest) ... ok (0.001s)
2022-11-23T02:26:27.0050029Z   test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s)
2022-11-23T02:26:27.0050729Z   test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:27.0051223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:27.0051849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:27.0052325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:27.0052908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:26:27.0053338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:26:27.0053911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:26:27.0054455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:26:27.0054741Z world
2022-11-23T02:26:27.0054950Z hello
2022-11-23T02:26:27.0055210Z Closing process 59828 via signal SIGTERM
2022-11-23T02:26:27.0055461Z ok (2.412s)
2022-11-23T02:26:27.0055755Z   test_from_str_bad_input (__main__.StdTest) ... ok (0.001s)
2022-11-23T02:26:27.0056096Z   test_from_value (__main__.StdTest) ... ok (0.001s)
2022-11-23T02:26:27.0056431Z   test_from_value_map (__main__.StdTest) ... ok (0.001s)
2022-11-23T02:26:27.0056613Z 
2022-11-23T02:26:27.0056893Z ----------------------------------------------------------------------
2022-11-23T02:26:27.0057227Z Ran 38 tests in 27.434s
2022-11-23T02:26:27.0057391Z 
2022-11-23T02:26:27.0057486Z OK
2022-11-23T02:26:27.0057621Z 
2022-11-23T02:26:27.0057729Z Generating XML reports...
2022-11-23T02:26:27.0058382Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20221123022559.xml
2022-11-23T02:26:27.0059261Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20221123022559.xml
2022-11-23T02:26:27.0060126Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20221123022559.xml
2022-11-23T02:26:27.0060920Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20221123022559.xml
2022-11-23T02:26:27.0061282Z 
2022-11-23T02:26:27.0061722Z ##[endgroup]
2022-11-23T02:26:27.0062404Z FINISHED PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_dnv4i9_j)
2022-11-23T02:26:27.0062816Z 
2022-11-23T02:26:27.0063083Z Running distributed/fsdp/test_fsdp_meta ... [2022-11-23 02:26:26.994979]
2022-11-23T02:26:27.0063736Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:26:26.995363]
2022-11-23T02:27:01.5621763Z 
2022-11-23T02:27:01.5622293Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_meta
2022-11-23T02:27:01.5623333Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_1emsnc6q)
2022-11-23T02:27:01.5623954Z 
2022-11-23T02:27:01.5625976Z Running tests...
2022-11-23T02:27:01.5626529Z ----------------------------------------------------------------------
2022-11-23T02:27:01.5627597Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta
2022-11-23T02:27:01.5628119Z   test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:01.5628584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59928
2022-11-23T02:27:01.5629046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59929
2022-11-23T02:27:01.5629684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5630153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5630720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5631213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5631796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5632240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5632797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5635492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5635967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:01.5636728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:01.5637453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5638166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5638678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:01.5639167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:01.5639530Z dist init r=1, world=2
2022-11-23T02:27:01.5639770Z dist init r=0, world=2
2022-11-23T02:27:01.5640011Z ok (5.542s)
2022-11-23T02:27:01.5640482Z   test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T02:27:01.5641192Z   test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60007
2022-11-23T02:27:01.5641756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60008
2022-11-23T02:27:01.5642391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5642862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5643427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5643899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5644485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5644942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5645505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5645977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5646545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:01.5647057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:01.5647712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5648406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5648931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:01.5649395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:01.5649758Z dist init r=1, world=2
2022-11-23T02:27:01.5650016Z dist init r=0, world=2
2022-11-23T02:27:01.5650261Z ok (4.512s)
2022-11-23T02:27:01.5650864Z   test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60090
2022-11-23T02:27:01.5651437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60091
2022-11-23T02:27:01.5652057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5652511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5653073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5653547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5654214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5654665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5655227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5655698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5656158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:01.5656657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:01.5657299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5657994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5658524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:01.5658981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:01.5659341Z dist init r=1, world=2
2022-11-23T02:27:01.5659599Z dist init r=0, world=2
2022-11-23T02:27:01.5659844Z ok (4.411s)
2022-11-23T02:27:01.5660318Z   test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60173
2022-11-23T02:27:01.5660903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60174
2022-11-23T02:27:01.5661522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5661960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5662544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5663016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5663597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5664095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5664686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5665155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5665596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:01.5666097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:01.5667103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5667858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5668364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:01.5668845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:01.5669196Z dist init r=1, world=2
2022-11-23T02:27:01.5669453Z dist init r=0, world=2
2022-11-23T02:27:01.5669674Z ok (4.511s)
2022-11-23T02:27:01.5670165Z   test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60256
2022-11-23T02:27:01.5670744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60257
2022-11-23T02:27:01.5671345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5671924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5672502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5672976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5673539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5673982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5674552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5674997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5675453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:01.5675957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:01.5676617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5677292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5677815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:01.5678285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:01.5678644Z dist init r=1, world=2
2022-11-23T02:27:01.5678878Z dist init r=0, world=2
2022-11-23T02:27:01.5679121Z ok (4.411s)
2022-11-23T02:27:01.5679626Z   test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2022-11-23T02:27:01.5680340Z   test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T02:27:01.5681129Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T02:27:01.5681845Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T02:27:01.5682539Z   test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60339
2022-11-23T02:27:01.5683083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60340
2022-11-23T02:27:01.5683709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5684168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5684747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5685206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5685789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5686232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5686813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5687261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5687717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:01.5688292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:01.5688940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5689635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5690159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:01.5690627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:01.5690965Z dist init r=1, world=2
2022-11-23T02:27:01.5691220Z dist init r=0, world=2
2022-11-23T02:27:01.5691461Z ok (4.411s)
2022-11-23T02:27:01.5691915Z   test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60422
2022-11-23T02:27:01.5692482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60423
2022-11-23T02:27:01.5693097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5693550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5694116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5694588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5695167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:01.5695592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:01.5696167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:01.5696639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:01.5697093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:27:01.5697572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:27:01.5698295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5698995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:27:01.5699524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:01.5699980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:01.5700331Z dist init r=0, world=2
2022-11-23T02:27:01.5700583Z dist init r=1, world=2
2022-11-23T02:27:01.5700808Z ok (4.411s)
2022-11-23T02:27:01.5701294Z   test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T02:27:01.5701979Z   test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2022-11-23T02:27:01.5702341Z 
2022-11-23T02:27:01.5702619Z ----------------------------------------------------------------------
2022-11-23T02:27:01.5702935Z Ran 14 tests in 32.214s
2022-11-23T02:27:01.5703100Z 
2022-11-23T02:27:01.5703209Z OK (skipped=7)
2022-11-23T02:27:01.5703364Z 
2022-11-23T02:27:01.5703489Z Generating XML reports...
2022-11-23T02:27:01.5704086Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221123022628.xml
2022-11-23T02:27:01.5704455Z 
2022-11-23T02:27:01.5704887Z ##[endgroup]
2022-11-23T02:27:01.5705482Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_1emsnc6q)
2022-11-23T02:27:01.5705910Z 
2022-11-23T02:27:01.5706229Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks ... [2022-11-23 02:27:01.562374]
2022-11-23T02:27:01.5707297Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:27:01.562633]
2022-11-23T02:27:37.7896714Z 
2022-11-23T02:27:37.7897263Z Expand the folded group to see the log file of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2022-11-23T02:27:37.7898385Z ##[group]PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_xa9xjd2i)
2022-11-23T02:27:37.7900305Z 
2022-11-23T02:27:37.7900647Z Running tests...
2022-11-23T02:27:37.7904940Z ----------------------------------------------------------------------
2022-11-23T02:27:37.7905634Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks
2022-11-23T02:27:37.7906179Z   test_ddp_comm_hook_allreduce_hook (__main__.DistributedDataParallelCommHookTest)
2022-11-23T02:27:37.7906725Z This unit test verifies the ``allreduce`` hook registered case gives same result ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:27:37.7907601Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60540
2022-11-23T02:27:37.7908027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60541
2022-11-23T02:27:37.7911264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:37.7912189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:37.7913134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:37.7913820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:37.7915112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:37.7916019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:37.7916854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:37.7917352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:37.7917792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:37.7918290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:37.7918967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ehkgvvb
2022-11-23T02:27:37.7919512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ehkgvvb/_remote_module_non_scriptable.py
2022-11-23T02:27:37.7920057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mdztst3
2022-11-23T02:27:37.7920589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mdztst3/_remote_module_non_scriptable.py
2022-11-23T02:27:37.7921162Z ok (6.559s)
2022-11-23T02:27:37.7921554Z   test_ddp_comm_hook_fp16compress_hook (__main__.DistributedDataParallelCommHookTest)
2022-11-23T02:27:37.7922127Z This unit test verifies the ``fp16 compress`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60624
2022-11-23T02:27:37.7922647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60625
2022-11-23T02:27:37.7923275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:37.7923735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:37.7924464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:37.7924991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:37.7925555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:37.7925992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:37.7926560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:37.7927024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:37.7927442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:37.7927911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:37.7928409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpryolpctb
2022-11-23T02:27:37.7928940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpryolpctb/_remote_module_non_scriptable.py
2022-11-23T02:27:37.7929466Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8w_8y8dy
2022-11-23T02:27:37.7929999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8w_8y8dy/_remote_module_non_scriptable.py
2022-11-23T02:27:37.7930373Z ok (5.012s)
2022-11-23T02:27:37.7930712Z   test_ddp_comm_hook_noop_hook (__main__.DistributedDataParallelCommHookTest)
2022-11-23T02:27:37.7931293Z This unit test verifies the ``noop`` hook registered case and a subsequent allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60708
2022-11-23T02:27:37.7931841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60709
2022-11-23T02:27:37.7932438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:37.7932890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:37.7933465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:37.7933929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:37.7934560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:37.7935018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:37.7935593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:37.7936051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:37.7936471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:37.7936941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:37.7937443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0yi0454
2022-11-23T02:27:37.7937960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0yi0454/_remote_module_non_scriptable.py
2022-11-23T02:27:37.7938490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wk8la4q
2022-11-23T02:27:37.7939018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wk8la4q/_remote_module_non_scriptable.py
2022-11-23T02:27:37.7939394Z ok (5.013s)
2022-11-23T02:27:37.7939761Z   test_ddp_comm_hook_quantize_per_channel_hook (__main__.DistributedDataParallelCommHookTest)
2022-11-23T02:27:37.7940345Z This unit test verifies the ``quantize per channel`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60792
2022-11-23T02:27:37.7940884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60793
2022-11-23T02:27:37.7941556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:37.7942004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:37.7942573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:37.7943040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:37.7943599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:37.7944043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:37.7944608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:37.7945050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:37.7945486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:37.7945965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:37.7946462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76_3f_jq
2022-11-23T02:27:37.7947478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76_3f_jq/_remote_module_non_scriptable.py
2022-11-23T02:27:37.7948021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezg_ubk3
2022-11-23T02:27:37.7948545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezg_ubk3/_remote_module_non_scriptable.py
2022-11-23T02:27:37.7948919Z ok (4.912s)
2022-11-23T02:27:37.7949279Z   test_ddp_comm_hook_quantize_per_tensor_hook (__main__.DistributedDataParallelCommHookTest)
2022-11-23T02:27:37.7949862Z This unit test verifies the ``quantize per tensor`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60876
2022-11-23T02:27:37.7950443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60877
2022-11-23T02:27:37.7951050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:37.7951499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:37.7952177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:37.7952655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:37.7953220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:37.7953662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:37.7954225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:37.7954676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:37.7955111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:37.7955580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:37.7956079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvfxb295
2022-11-23T02:27:37.7956602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvfxb295/_remote_module_non_scriptable.py
2022-11-23T02:27:37.7957126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4u_s8om
2022-11-23T02:27:37.7957653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4u_s8om/_remote_module_non_scriptable.py
2022-11-23T02:27:37.7958025Z ok (5.013s)
2022-11-23T02:27:37.7958471Z   test_is_last_hook (__main__.DistributedDataParallelCommHookTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60960
2022-11-23T02:27:37.7959118Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60961
2022-11-23T02:27:37.7959736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:37.7960167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:37.7960738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:37.7961203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:37.7961774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:27:37.7962198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:27:37.7962761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:27:37.7963227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:27:37.7963642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:27:37.7964340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:27:37.7965177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppdyamfvr
2022-11-23T02:27:37.7965751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppdyamfvr/_remote_module_non_scriptable.py
2022-11-23T02:27:37.7966271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbtauty0o
2022-11-23T02:27:37.7966808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbtauty0o/_remote_module_non_scriptable.py
2022-11-23T02:27:37.7967185Z ok (7.417s)
2022-11-23T02:27:37.7967334Z 
2022-11-23T02:27:37.7967617Z ----------------------------------------------------------------------
2022-11-23T02:27:37.7967934Z Ran 6 tests in 33.927s
2022-11-23T02:27:37.7968096Z 
2022-11-23T02:27:37.7968187Z OK
2022-11-23T02:27:37.7968322Z 
2022-11-23T02:27:37.7968443Z Generating XML reports...
2022-11-23T02:27:37.7969200Z Generated XML report: test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20221123022703.xml
2022-11-23T02:27:37.7969889Z 
2022-11-23T02:27:37.7970290Z ##[endgroup]
2022-11-23T02:27:37.7970983Z FINISHED PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_xa9xjd2i)
2022-11-23T02:27:37.7971391Z 
2022-11-23T02:27:37.7971669Z Running distributed/fsdp/test_fsdp_exec_order ... [2022-11-23 02:27:37.790073]
2022-11-23T02:27:37.7972340Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:27:37.790442]
2022-11-23T02:28:17.1433824Z 
2022-11-23T02:28:17.1434569Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_exec_order
2022-11-23T02:28:17.1436898Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_5_9mn2_b)
2022-11-23T02:28:17.1437635Z 
2022-11-23T02:28:17.1437752Z Running tests...
2022-11-23T02:28:17.1438322Z ----------------------------------------------------------------------
2022-11-23T02:28:17.1438898Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order
2022-11-23T02:28:17.1439425Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder)
2022-11-23T02:28:17.1440041Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:28:17.1440550Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61079
2022-11-23T02:28:17.1441301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61080
2022-11-23T02:28:17.1441922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1442373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1442958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1443426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1443991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1444453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1445027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1445493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1445938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:17.1446443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:17.1447101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1449877Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1450420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:17.1450890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:17.1452239Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1453103Z   warnings.warn(
2022-11-23T02:28:17.1454415Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1455204Z   warnings.warn(
2022-11-23T02:28:17.1455450Z dist init r=0, world=2
2022-11-23T02:28:17.1455679Z dist init r=1, world=2
2022-11-23T02:28:17.1455919Z ok (6.041s)
2022-11-23T02:28:17.1456309Z   test_invalid_first_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder)
2022-11-23T02:28:17.1457020Z Tests that FSDP errors if the all-gather order differs across ranks ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61158
2022-11-23T02:28:17.1457565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61159
2022-11-23T02:28:17.1458174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1458626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1459180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1459647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1460228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1460733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1461309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1461771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1462222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:17.1462702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:17.1463359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1464046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1464564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:17.1465024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:17.1466280Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1467434Z   warnings.warn(
2022-11-23T02:28:17.1468602Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1469378Z   warnings.warn(
2022-11-23T02:28:17.1469606Z dist init r=1, world=2
2022-11-23T02:28:17.1469857Z dist init r=0, world=2
2022-11-23T02:28:17.1470093Z ok (4.412s)
2022-11-23T02:28:17.1470589Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-11-23T02:28:17.1471361Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61237
2022-11-23T02:28:17.1471898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61238
2022-11-23T02:28:17.1472502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1472934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1473508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1473975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1474555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1474981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1475549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1476008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1476446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:17.1476941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:17.1477684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1478372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1478873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:17.1479346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:17.1480602Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1481381Z   warnings.warn(
2022-11-23T02:28:17.1482531Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1483284Z   warnings.warn(
2022-11-23T02:28:17.1483531Z dist init r=0, world=2
2022-11-23T02:28:17.1483780Z dist init r=1, world=2
2022-11-23T02:28:17.1483998Z ok (4.413s)
2022-11-23T02:28:17.1484412Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_FULL_SHARD_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-11-23T02:28:17.1485163Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61320
2022-11-23T02:28:17.1485706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61321
2022-11-23T02:28:17.1486297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1486802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1487385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1487854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1488411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1488849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1489422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1489872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1490323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:17.1490817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:17.1491477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1492147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1492666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:17.1493139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:17.1494395Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1495214Z   warnings.warn(
2022-11-23T02:28:17.1496373Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1497127Z   warnings.warn(
2022-11-23T02:28:17.1497429Z dist init r=0, world=2
2022-11-23T02:28:17.1497678Z dist init r=1, world=2
2022-11-23T02:28:17.1497895Z ok (4.413s)
2022-11-23T02:28:17.1498315Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_1 (__main__.TestFSDPExecOrder)
2022-11-23T02:28:17.1499066Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61403
2022-11-23T02:28:17.1499609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61404
2022-11-23T02:28:17.1500196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1500644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1501212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1501680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1502243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1502683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1503310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1503762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1504211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:17.1504704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:17.1505360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1506025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1506550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:17.1507288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:17.1508563Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1509320Z   warnings.warn(
2022-11-23T02:28:17.1510451Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1511319Z   warnings.warn(
2022-11-23T02:28:17.1511569Z dist init r=0, world=2
2022-11-23T02:28:17.1511818Z dist init r=1, world=2
2022-11-23T02:28:17.1512036Z ok (4.412s)
2022-11-23T02:28:17.1512459Z   test_invalid_later_iter_order_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_iters_before_path_change_3 (__main__.TestFSDPExecOrder)
2022-11-23T02:28:17.1513213Z Tests that FSDP warns the user if the all-gather order changes after ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61486
2022-11-23T02:28:17.1513735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61487
2022-11-23T02:28:17.1514342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1514791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1515362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1515814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1516389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1516829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1517380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1517837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1518290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:17.1518789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:17.1519432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1520187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1520719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:17.1521189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:17.1522437Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1523211Z   warnings.warn(
2022-11-23T02:28:17.1524361Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1525126Z   warnings.warn(
2022-11-23T02:28:17.1525370Z dist init r=0, world=2
2022-11-23T02:28:17.1525600Z dist init r=1, world=2
2022-11-23T02:28:17.1525835Z ok (4.412s)
2022-11-23T02:28:17.1526314Z   test_train_eval_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61569
2022-11-23T02:28:17.1526932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61570
2022-11-23T02:28:17.1527539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1527987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1528557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1529005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1529579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1530021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1530571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1531038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1531490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:17.1531985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:17.1532627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1533313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1533829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:17.1534297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:17.1535595Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1536382Z   warnings.warn(
2022-11-23T02:28:17.1537528Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1538278Z   warnings.warn(
2022-11-23T02:28:17.1538527Z dist init r=0, world=2
2022-11-23T02:28:17.1538758Z dist init r=1, world=2
2022-11-23T02:28:17.1538994Z ok (4.412s)
2022-11-23T02:28:17.1539477Z   test_train_eval_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestFSDPExecOrder) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61652
2022-11-23T02:28:17.1540035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61653
2022-11-23T02:28:17.1540647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1541095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1541669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1542119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1542691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:28:17.1543198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:28:17.1543752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:28:17.1544214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:28:17.1544667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:28:17.1545160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:28:17.1545802Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1546487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:28:17.1547288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:28:17.1547765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:28:17.1549025Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1549825Z   warnings.warn(
2022-11-23T02:28:17.1550975Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:28:17.1551747Z   warnings.warn(
2022-11-23T02:28:17.1551992Z dist init r=1, world=2
2022-11-23T02:28:17.1552221Z dist init r=0, world=2
2022-11-23T02:28:17.1552459Z ok (4.512s)
2022-11-23T02:28:17.1552606Z 
2022-11-23T02:28:17.1552987Z ----------------------------------------------------------------------
2022-11-23T02:28:17.1553308Z Ran 8 tests in 37.027s
2022-11-23T02:28:17.1553466Z 
2022-11-23T02:28:17.1553559Z OK
2022-11-23T02:28:17.1553690Z 
2022-11-23T02:28:17.1553815Z Generating XML reports...
2022-11-23T02:28:17.1554407Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20221123022739.xml
2022-11-23T02:28:17.1554766Z 
2022-11-23T02:28:17.1555151Z ##[endgroup]
2022-11-23T02:28:17.1555762Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_exec_order (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_exec_order_5_9mn2_b)
2022-11-23T02:28:17.1556129Z 
2022-11-23T02:28:17.1556393Z Running distributed/_tensor/test_dtensor ... [2022-11-23 02:28:17.143547]
2022-11-23T02:28:17.1557054Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:28:17.143816]
2022-11-23T02:29:01.8043483Z 
2022-11-23T02:29:01.8044529Z Expand the folded group to see the log file of distributed/_tensor/test_dtensor
2022-11-23T02:29:01.8045733Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_dtensor_bhqc2yd0)
2022-11-23T02:29:01.8046093Z 
2022-11-23T02:29:01.8046204Z Running tests...
2022-11-23T02:29:01.8052473Z ----------------------------------------------------------------------
2022-11-23T02:29:01.8053237Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_dtensor
2022-11-23T02:29:01.8054011Z   test_device_mesh_nd (__main__.DTensorMeshTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:01.8054484Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61770
2022-11-23T02:29:01.8054941Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61771
2022-11-23T02:29:01.8055391Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 61772
2022-11-23T02:29:01.8055836Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 61773
2022-11-23T02:29:01.8056270Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 61774
2022-11-23T02:29:01.8056695Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 61775
2022-11-23T02:29:01.8057130Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 61776
2022-11-23T02:29:01.8057563Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 61777
2022-11-23T02:29:01.8058213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8058655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8059231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8059714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8060298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8060725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8061306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8061768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8062329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8062780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8063357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8063942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8064523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8064981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8065544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8066017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8066568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8067441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8068026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8068494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8069058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8069502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8070082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8070525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8071106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8071664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8072546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8073390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8074538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8075355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8076443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8077473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8078000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:29:01.8078473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:01.8078929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:29:01.8079392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T02:29:01.8079852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:01.8080323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:29:01.8080765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T02:29:01.8081211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:29:01.8081603Z skip: Need at least 8 CUDA devices (4.134s)
2022-11-23T02:29:01.8082047Z   test_dtensor_2d_mesh (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62042
2022-11-23T02:29:01.8082561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62043
2022-11-23T02:29:01.8083008Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62044
2022-11-23T02:29:01.8083456Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62045
2022-11-23T02:29:01.8084014Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 62046
2022-11-23T02:29:01.8084471Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 62047
2022-11-23T02:29:01.8084908Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 62048
2022-11-23T02:29:01.8085325Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 62049
2022-11-23T02:29:01.8085952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8086406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8086969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8087401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8087977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8088604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8089196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8089643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8090220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8090670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8091221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8091767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8092344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8092787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8093344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8093810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8094384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8094825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8095378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8095845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8096420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8096845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8097422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8097884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8098460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8098880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8099449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8099913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8100469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8100909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8101543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8102018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8102434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T02:29:01.8102904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:29:01.8103369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:29:01.8103832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:29:01.8104285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:29:01.8104739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:01.8105195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:01.8105641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T02:29:01.8106025Z skip: Need at least 8 CUDA devices (2.615s)
2022-11-23T02:29:01.8106519Z   test_dtensor_api_device_mesh_context_manager (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62314
2022-11-23T02:29:01.8107617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62315
2022-11-23T02:29:01.8108054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62316
2022-11-23T02:29:01.8108489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62317
2022-11-23T02:29:01.8109048Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 62318
2022-11-23T02:29:01.8109470Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 62319
2022-11-23T02:29:01.8109905Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 62320
2022-11-23T02:29:01.8110344Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 62321
2022-11-23T02:29:01.8110970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8111403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8111977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8112445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8113002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8113449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8114019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8114485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8115045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8115496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8116064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8116527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8117082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8117531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8118100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8118539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8119191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8119648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8120221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8120662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8121240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8121685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8122233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8122690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8123271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8123717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8124266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8124728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8125302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8125808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8126360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8126823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8127262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T02:29:01.8127722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:29:01.8128188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:29:01.8128644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:01.8129107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T02:29:01.8129554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:01.8130010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:29:01.8130469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:29:01.8130837Z skip: Need at least 8 CUDA devices (2.615s)
2022-11-23T02:29:01.8131340Z   test_dtensor_device_mesh_device_conversion (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62586
2022-11-23T02:29:01.8131879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62587
2022-11-23T02:29:01.8132324Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62588
2022-11-23T02:29:01.8132747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62589
2022-11-23T02:29:01.8133176Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 62590
2022-11-23T02:29:01.8133617Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 62591
2022-11-23T02:29:01.8134042Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 62592
2022-11-23T02:29:01.8134477Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 62593
2022-11-23T02:29:01.8135083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8135596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8136170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8136635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8137216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8137661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8138213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8138678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8139254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8139682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8140253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8140714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8141287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8141708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8142277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8142809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8143371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8143807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8144379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8144840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8145395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8145837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8146405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8147462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8148057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8148497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8149106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8149553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8150130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8150572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8151141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8151583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8152025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:01.8152496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T02:29:01.8153045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:01.8153523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:29:01.8153979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:29:01.8154441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:29:01.8154881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T02:29:01.8155333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:29:01.8155734Z skip: Need at least 8 CUDA devices (2.614s)
2022-11-23T02:29:01.8156214Z   test_dtensor_spec_local_shard_offset (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62858
2022-11-23T02:29:01.8156728Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62859
2022-11-23T02:29:01.8157250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62860
2022-11-23T02:29:01.8157881Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62861
2022-11-23T02:29:01.8158342Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 62862
2022-11-23T02:29:01.8158823Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 62863
2022-11-23T02:29:01.8159456Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 62864
2022-11-23T02:29:01.8177185Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 62865
2022-11-23T02:29:01.8178039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8178505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8179064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8179511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8180094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8180550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8181142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8181612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8182187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8182619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8183188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8183658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8184239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8184664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8185231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8185695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8186250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8186698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8187921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8188384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8189059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8189527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8190106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8190549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8191117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8191568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8192134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8192572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8193149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8193594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8194157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8194596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8195034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:01.8195505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T02:29:01.8196049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2022-11-23T02:29:01.8196512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2022-11-23T02:29:01.8196985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:01.8197450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2022-11-23T02:29:01.8197894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T02:29:01.8198353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2022-11-23T02:29:01.8198741Z skip: Need at least 8 CUDA devices (2.615s)
2022-11-23T02:29:01.8199183Z   test_dtensor_constructor (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63130
2022-11-23T02:29:01.8199703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63131
2022-11-23T02:29:01.8200325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8200774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8201333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8201799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8202374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8202815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8203366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8203827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8204265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:01.8204739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:01.8205228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:01.8205775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:01.8206446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8207119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8207516Z ok (3.911s)
2022-11-23T02:29:01.8207933Z   test_dtensor_properties (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63209
2022-11-23T02:29:01.8208432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63210
2022-11-23T02:29:01.8209044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8209494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8210072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8210529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8211106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8211549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8212119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8212637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8213073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:01.8213560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:01.8214031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:01.8214514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:01.8215176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8215864Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8216239Z ok (3.910s)
2022-11-23T02:29:01.8216669Z   test_dtensor_spec_read_only_after_set (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63288
2022-11-23T02:29:01.8217202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63289
2022-11-23T02:29:01.8217812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8218247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8218827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8219295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8219849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8220291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8220855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8221321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8221737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:01.8222230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:01.8222778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:01.8223252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:01.8223913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8224601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8224995Z ok (4.010s)
2022-11-23T02:29:01.8225386Z   test_dtensor_stride (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63367
2022-11-23T02:29:01.8225888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63368
2022-11-23T02:29:01.8226498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8227494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8228077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8228543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8229117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8229538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8230108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8230696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8231130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:01.8231587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:01.8232063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:01.8232563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:01.8233212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8233901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8234290Z ok (4.011s)
2022-11-23T02:29:01.8234694Z   test_from_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63446
2022-11-23T02:29:01.8235172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63447
2022-11-23T02:29:01.8235783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8236230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8236805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8237256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8237832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8238272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8238824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8239293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8239731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:01.8240301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:01.8240781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:01.8241270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:01.8241931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8242615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8242997Z ok (3.912s)
2022-11-23T02:29:01.8243419Z   test_from_local_then_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63529
2022-11-23T02:29:01.8243929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63530
2022-11-23T02:29:01.8244525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8244975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8245550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8246014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8246571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8247012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8247650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8248091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8248530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:01.8249070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:01.8249555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:01.8250014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:01.8250672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8251358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8251753Z ok (3.911s)
2022-11-23T02:29:01.8252132Z   test_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63612
2022-11-23T02:29:01.8252628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63613
2022-11-23T02:29:01.8253243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8253673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8254247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8254718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8255293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:01.8255714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:01.8256285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:01.8256746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:01.8257224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:01.8257725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:01.8258209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:01.8258692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:01.8259332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8260017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:01.8260418Z ok (4.011s)
2022-11-23T02:29:01.8260570Z 
2022-11-23T02:29:01.8260839Z ----------------------------------------------------------------------
2022-11-23T02:29:01.8261156Z Ran 12 tests in 42.269s
2022-11-23T02:29:01.8261319Z 
2022-11-23T02:29:01.8261429Z OK (skipped=5)
2022-11-23T02:29:01.8261587Z 
2022-11-23T02:29:01.8261711Z Generating XML reports...
2022-11-23T02:29:01.8262257Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20221123022819.xml
2022-11-23T02:29:01.8262980Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20221123022819.xml
2022-11-23T02:29:01.8263314Z 
2022-11-23T02:29:01.8263672Z ##[endgroup]
2022-11-23T02:29:01.8264238Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_dtensor_bhqc2yd0)
2022-11-23T02:29:01.8264662Z 
2022-11-23T02:29:01.8264932Z Running distributed/algorithms/test_join ... [2022-11-23 02:29:01.804820]
2022-11-23T02:29:01.8265605Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:29:01.805104]
2022-11-23T02:29:50.7720222Z 
2022-11-23T02:29:50.7720855Z Expand the folded group to see the log file of distributed/algorithms/test_join
2022-11-23T02:29:50.7722282Z ##[group]PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_aev58lub)
2022-11-23T02:29:50.7725559Z 
2022-11-23T02:29:50.7725933Z Running tests...
2022-11-23T02:29:50.7726757Z ----------------------------------------------------------------------
2022-11-23T02:29:50.7727862Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join
2022-11-23T02:29:50.7728293Z   test_join_kwargs (__main__.TestJoin)
2022-11-23T02:29:50.7728727Z Tests passing keyword arguments to the context manager. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:29:50.7729188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63730
2022-11-23T02:29:50.7729644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63731
2022-11-23T02:29:50.7730556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7731024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7731609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7732090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7732668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7733100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7733688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7734356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7735122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:50.7735604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:50.7736093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:50.7736592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:50.7737525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7738330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7738720Z ok (6.547s)
2022-11-23T02:29:50.7739021Z   test_multiple_joinable_disable (__main__.TestJoin)
2022-11-23T02:29:50.7739509Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63810
2022-11-23T02:29:50.7740011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63811
2022-11-23T02:29:50.7740629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7741083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7741664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7742117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7742699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7743297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7743861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7744334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7744777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:50.7745274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:50.7745747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:50.7746234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:50.7747282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7748041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7748419Z ok (5.014s)
2022-11-23T02:29:50.7748701Z   test_multiple_joinables (__main__.TestJoin)
2022-11-23T02:29:50.7749326Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63890
2022-11-23T02:29:50.7749847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63891
2022-11-23T02:29:50.7750458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7750909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7751487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7751945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7752530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7752977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7753641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7754124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7754567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:50.7755060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:50.7755528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:50.7756016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:50.7756688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7757380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7757756Z ok (5.013s)
2022-11-23T02:29:50.7758051Z   test_multiple_joinables_throw (__main__.TestJoin)
2022-11-23T02:29:50.7758535Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63970
2022-11-23T02:29:50.7759037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63971
2022-11-23T02:29:50.7759652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7760107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7760683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7761236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7761821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7762272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7762826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7763293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7763733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:50.7764229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:50.7764697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:50.7765189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:50.7765857Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7766553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7766932Z ok (5.013s)
2022-11-23T02:29:50.7767207Z   test_single_joinable (__main__.TestJoin)
2022-11-23T02:29:50.7767818Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64050
2022-11-23T02:29:50.7768341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64051
2022-11-23T02:29:50.7768949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7769404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7769980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7770435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7771083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7771535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7772096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7772565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7773004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:50.7773499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:50.7774041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:50.7774509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:50.7775174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7775862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7776255Z ok (5.013s)
2022-11-23T02:29:50.7776523Z   test_single_joinable_disable (__main__.TestJoin)
2022-11-23T02:29:50.7776998Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64130
2022-11-23T02:29:50.7777511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64131
2022-11-23T02:29:50.7778182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7778632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7779206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7779675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7780237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7780688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7781258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7781706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7782142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:50.7782637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:50.7783121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:50.7783588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:50.7784250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7784946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7785336Z ok (5.013s)
2022-11-23T02:29:50.7785609Z   test_single_joinable_main_hooks (__main__.TestJoin)
2022-11-23T02:29:50.7786082Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64210
2022-11-23T02:29:50.7786599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64211
2022-11-23T02:29:50.7787494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7787946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7788614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7789098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7789664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7790110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7790682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7791135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7791576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:50.7792070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:50.7792563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:50.7793029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:50.7793690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7794380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7794777Z ok (5.033s)
2022-11-23T02:29:50.7795048Z   test_single_joinable_post_hooks (__main__.TestJoin)
2022-11-23T02:29:50.7795744Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64290
2022-11-23T02:29:50.7796264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64291
2022-11-23T02:29:50.7796851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7797308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7797884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7798352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7798912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7799362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7799931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7800383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7800821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:50.7801317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:50.7801804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:50.7802269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:50.7802935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7803626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7804028Z ok (5.013s)
2022-11-23T02:29:50.7804292Z   test_single_joinable_throw (__main__.TestJoin)
2022-11-23T02:29:50.7804766Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64370
2022-11-23T02:29:50.7805280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64371
2022-11-23T02:29:50.7805935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7806395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7806975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7807447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7808006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:29:50.7808454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:29:50.7809027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:29:50.7809475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:29:50.7809921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:29:50.7810412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:29:50.7810898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:29:50.7811359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:29:50.7812023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7812717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:29:50.7813196Z ok (5.013s)
2022-11-23T02:29:50.7813325Z 
2022-11-23T02:29:50.7813601Z ----------------------------------------------------------------------
2022-11-23T02:29:50.7813935Z Ran 9 tests in 46.672s
2022-11-23T02:29:50.7814101Z 
2022-11-23T02:29:50.7814196Z OK
2022-11-23T02:29:50.7814335Z 
2022-11-23T02:29:50.7814441Z Generating XML reports...
2022-11-23T02:29:50.7815013Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20221123022903.xml
2022-11-23T02:29:50.7815348Z 
2022-11-23T02:29:50.7815708Z ##[endgroup]
2022-11-23T02:29:50.7816287Z FINISHED PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_aev58lub)
2022-11-23T02:29:50.7816647Z 
2022-11-23T02:29:50.7816921Z Running distributed/fsdp/test_fsdp_grad_acc ... [2022-11-23 02:29:50.772133]
2022-11-23T02:29:50.7817616Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:29:50.772398]
2022-11-23T02:30:48.8572156Z 
2022-11-23T02:30:48.8574688Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_grad_acc
2022-11-23T02:30:48.8575647Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_wu76p_tk)
2022-11-23T02:30:48.8576022Z 
2022-11-23T02:30:48.8576156Z Running tests...
2022-11-23T02:30:48.8577322Z ----------------------------------------------------------------------
2022-11-23T02:30:48.8577920Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc
2022-11-23T02:30:48.8578587Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T02:30:48.8579236Z Tests gradient accumulation. ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:30:48.8579716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64485
2022-11-23T02:30:48.8581149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64486
2022-11-23T02:30:48.8582035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8582508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8585172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8585697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8586294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8587104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8587708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8588187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8588682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:48.8589186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:48.8589849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8590529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8591057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:48.8591740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:48.8593031Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:48.8593886Z   warnings.warn(
2022-11-23T02:30:48.8595028Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:48.8595816Z   warnings.warn(
2022-11-23T02:30:48.8596068Z dist init r=1, world=2
2022-11-23T02:30:48.8596330Z dist init r=0, world=2
2022-11-23T02:30:48.8596551Z ok (6.814s)
2022-11-23T02:30:48.8597091Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T02:30:48.8597769Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64568
2022-11-23T02:30:48.8598294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64569
2022-11-23T02:30:48.8598897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8599360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8599941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8600413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8601073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8601537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8602113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8602563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8603016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:48.8603514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:48.8604177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8604852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8605384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:48.8605856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:48.8607132Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:48.8607971Z   warnings.warn(
2022-11-23T02:30:48.8609107Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:48.8609879Z   warnings.warn(
2022-11-23T02:30:48.8610230Z dist init r=0, world=2
2022-11-23T02:30:48.8610482Z dist init r=1, world=2
2022-11-23T02:30:48.8610723Z ok (5.013s)
2022-11-23T02:30:48.8611243Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T02:30:48.8611920Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64651
2022-11-23T02:30:48.8612414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64652
2022-11-23T02:30:48.8613034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8613471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8614048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8614518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8615097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8615522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8616091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8616561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8616996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:48.8617554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:48.8618225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8618913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8619420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:48.8619894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:48.8621162Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:48.8621928Z   warnings.warn(
2022-11-23T02:30:48.8623078Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:48.8623914Z   warnings.warn(
2022-11-23T02:30:48.8624146Z dist init r=1, world=2
2022-11-23T02:30:48.8624399Z dist init r=0, world=2
2022-11-23T02:30:48.8624640Z ok (5.113s)
2022-11-23T02:30:48.8625156Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T02:30:48.8625824Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64734
2022-11-23T02:30:48.8626318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64735
2022-11-23T02:30:48.8627406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8627873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8628465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8628943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8629507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8629957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8630533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8630999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8631435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:48.8631927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:48.8632592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8633269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8633797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:48.8634367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:48.8634739Z dist init r=1, world=2
2022-11-23T02:30:48.8634974Z dist init r=0, world=2
2022-11-23T02:30:48.8635213Z ok (3.911s)
2022-11-23T02:30:48.8635746Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T02:30:48.8636391Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64813
2022-11-23T02:30:48.8636882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64814
2022-11-23T02:30:48.8637506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8637960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8638520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8638997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8639581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8640024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8640584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8641051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8641594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:48.8642072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:48.8642740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8643436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8643961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:48.8644417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:48.8644765Z dist init r=1, world=2
2022-11-23T02:30:48.8645019Z dist init r=0, world=2
2022-11-23T02:30:48.8645238Z ok (3.911s)
2022-11-23T02:30:48.8645774Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T02:30:48.8646447Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64892
2022-11-23T02:30:48.8646989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64893
2022-11-23T02:30:48.8647590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8648043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8648619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8649089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8649652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8650109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8650681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8651191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8651661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:48.8652161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:48.8652820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8653492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8654014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:48.8654488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:48.8654845Z dist init r=1, world=2
2022-11-23T02:30:48.8655079Z dist init r=0, world=2
2022-11-23T02:30:48.8655319Z ok (3.911s)
2022-11-23T02:30:48.8655855Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T02:30:48.8656504Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64971
2022-11-23T02:30:48.8656998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64972
2022-11-23T02:30:48.8657608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8658124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8658683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8659150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8659729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8660154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8660727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8661189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8661642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:48.8662120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:48.8662787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8663475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8664001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:48.8664451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:48.8665714Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:48.8666494Z   warnings.warn(
2022-11-23T02:30:48.8668116Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:48.8668906Z   warnings.warn(
2022-11-23T02:30:48.8669137Z dist init r=1, world=2
2022-11-23T02:30:48.8669389Z dist init r=0, world=2
2022-11-23T02:30:48.8669629Z ok (5.113s)
2022-11-23T02:30:48.8670137Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T02:30:48.8670810Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65054
2022-11-23T02:30:48.8671303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65055
2022-11-23T02:30:48.8671922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8672358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8672931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8673404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8673985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8674410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8675084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8675550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8675986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:48.8676488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:48.8677143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8677831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8678333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:48.8678806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:48.8680077Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:48.8680843Z   warnings.warn(
2022-11-23T02:30:48.8681999Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:48.8682756Z   warnings.warn(
2022-11-23T02:30:48.8683004Z dist init r=0, world=2
2022-11-23T02:30:48.8683255Z dist init r=1, world=2
2022-11-23T02:30:48.8683477Z ok (5.013s)
2022-11-23T02:30:48.8684070Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=False)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T02:30:48.8684748Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65137
2022-11-23T02:30:48.8685241Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65138
2022-11-23T02:30:48.8685837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8686290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8686867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8687348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8687912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8688364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8688937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8689386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8689839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:48.8690334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:48.8690991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8691739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8692263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:48.8692739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:48.8694000Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:48.8694771Z   warnings.warn(
2022-11-23T02:30:48.8695896Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:30:48.8696669Z   warnings.warn(
2022-11-23T02:30:48.8696918Z dist init r=0, world=2
2022-11-23T02:30:48.8697167Z dist init r=1, world=2
2022-11-23T02:30:48.8697385Z ok (5.113s)
2022-11-23T02:30:48.8697912Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestGradAcc)
2022-11-23T02:30:48.8698578Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65220
2022-11-23T02:30:48.8699058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65221
2022-11-23T02:30:48.8699672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8700118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8700747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8701206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8701793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8702237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8702807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8703259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8703714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:48.8704213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:48.8704855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8705553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8706078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:48.8706550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:48.8707221Z dist init r=1, world=2
2022-11-23T02:30:48.8707478Z dist init r=0, world=2
2022-11-23T02:30:48.8707814Z ok (4.011s)
2022-11-23T02:30:48.8708324Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestGradAcc)
2022-11-23T02:30:48.8708993Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65299
2022-11-23T02:30:48.8709492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65300
2022-11-23T02:30:48.8710113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8710549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8711125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8711594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8712181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8712605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8713178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8713651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8714087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:48.8714586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:48.8715242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8715933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8716444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:48.8716918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:48.8717273Z dist init r=0, world=2
2022-11-23T02:30:48.8717505Z dist init r=1, world=2
2022-11-23T02:30:48.8717745Z ok (3.911s)
2022-11-23T02:30:48.8718355Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_cpu_offload_CPUOffload(offload_params=True)_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestGradAcc)
2022-11-23T02:30:48.8719035Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65378
2022-11-23T02:30:48.8719508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65379
2022-11-23T02:30:48.8720120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8720579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8721157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8721604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8722191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:30:48.8722635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:30:48.8723187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:30:48.8723654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:30:48.8724106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:30:48.8724673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:30:48.8725314Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8726001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:30:48.8726523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:30:48.8726992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:30:48.8727332Z dist init r=0, world=2
2022-11-23T02:30:48.8727585Z dist init r=1, world=2
2022-11-23T02:30:48.8727824Z ok (3.911s)
2022-11-23T02:30:48.8727955Z 
2022-11-23T02:30:48.8728229Z ----------------------------------------------------------------------
2022-11-23T02:30:48.8728561Z Ran 12 tests in 55.748s
2022-11-23T02:30:48.8728726Z 
2022-11-23T02:30:48.8728826Z OK
2022-11-23T02:30:48.8728959Z 
2022-11-23T02:30:48.8729065Z Generating XML reports...
2022-11-23T02:30:48.8729641Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20221123022952.xml
2022-11-23T02:30:48.8729977Z 
2022-11-23T02:30:48.8730373Z ##[endgroup]
2022-11-23T02:30:48.8730965Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_wu76p_tk)
2022-11-23T02:30:48.8731329Z 
2022-11-23T02:30:48.8731592Z Running distributed/fsdp/test_fsdp_misc ... [2022-11-23 02:30:48.857502]
2022-11-23T02:30:48.8732268Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:30:48.857871]
2022-11-23T02:31:58.5637860Z 
2022-11-23T02:31:58.5638253Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_misc
2022-11-23T02:31:58.5642246Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_misc_rdof6abn)
2022-11-23T02:31:58.5642662Z 
2022-11-23T02:31:58.5642763Z Running tests...
2022-11-23T02:31:58.5643282Z ----------------------------------------------------------------------
2022-11-23T02:31:58.5644059Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_misc
2022-11-23T02:31:58.5644540Z   test_cpu_init_with_sync_module_states (__main__.TestFSDPMisc)
2022-11-23T02:31:58.5644984Z Tests that passing ``sync_module_states=True`` raises an error for ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:31:58.5645522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65492
2022-11-23T02:31:58.5645980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65493
2022-11-23T02:31:58.5646604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5647065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5647653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5648127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5648701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5649163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5649753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5650213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5650653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5651152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5651945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5652650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5653181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5653659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5654942Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:31:58.5655734Z   warnings.warn(
2022-11-23T02:31:58.5656904Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:31:58.5657675Z   warnings.warn(
2022-11-23T02:31:58.5657903Z dist init r=1, world=2
2022-11-23T02:31:58.5658163Z dist init r=0, world=2
2022-11-23T02:31:58.5658408Z ok (5.524s)
2022-11-23T02:31:58.5658669Z   test_device_id_auto_wrap (__main__.TestFSDPMisc)
2022-11-23T02:31:58.5659164Z Tests that ``auto_wrap_policy`` propagates ``device_id`` to all ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65571
2022-11-23T02:31:58.5659710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65572
2022-11-23T02:31:58.5660333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5660772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5661401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5661894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5662467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5662912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5663474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5663950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5664390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5664884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5665547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5666242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5666754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5667673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5668027Z dist init r=1, world=2
2022-11-23T02:31:58.5668263Z dist init r=0, world=2
2022-11-23T02:31:58.5668614Z ok (3.911s)
2022-11-23T02:31:58.5669042Z   test_fsdp_cpu_init_stays_on_cpu (__main__.TestFSDPMisc)
2022-11-23T02:31:58.5669538Z Tests that passing a CPU module to FSDP preserves that the wrapped ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65650
2022-11-23T02:31:58.5670084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65651
2022-11-23T02:31:58.5670714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5671168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5671727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5672205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5672778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5673228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5673785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5674248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5674706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5675186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5675861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5676539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5677061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5677524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5677876Z dist init r=1, world=2
2022-11-23T02:31:58.5678132Z dist init r=0, world=2
2022-11-23T02:31:58.5678359Z ok (4.412s)
2022-11-23T02:31:58.5678661Z   test_fsdp_device_id_cpu_offload (__main__.TestFSDPMisc)
2022-11-23T02:31:58.5679407Z Ensures that even if device_id is specified but we have ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65733
2022-11-23T02:31:58.5679946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65734
2022-11-23T02:31:58.5680551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5681006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5681580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5682039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5682622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5683078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5683649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5684095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5684552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5685049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5685713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5686465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5686990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5687464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5687809Z dist init r=1, world=2
2022-11-23T02:31:58.5688063Z dist init r=0, world=2
2022-11-23T02:31:58.5688308Z ok (3.911s)
2022-11-23T02:31:58.5688598Z   test_fsdp_device_id_use_index_False (__main__.TestFSDPMisc)
2022-11-23T02:31:58.5689076Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65812
2022-11-23T02:31:58.5689576Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65813
2022-11-23T02:31:58.5690187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5690626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5691197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5691665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5692257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5692685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5693258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5693724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5694162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5694659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5695318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5696007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5696580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5697058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5697412Z dist init r=0, world=2
2022-11-23T02:31:58.5697645Z dist init r=1, world=2
2022-11-23T02:31:58.5697887Z ok (3.912s)
2022-11-23T02:31:58.5698196Z   test_fsdp_device_id_use_index_True (__main__.TestFSDPMisc)
2022-11-23T02:31:58.5698671Z Tests the FSDP ``device_id`` argument: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65891
2022-11-23T02:31:58.5699153Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65892
2022-11-23T02:31:58.5699769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5700220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5700779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5701242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5701821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5702259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5702809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5703271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5703794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5704274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5704934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5705621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5706152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5706608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5707282Z dist init r=0, world=2
2022-11-23T02:31:58.5707538Z dist init r=1, world=2
2022-11-23T02:31:58.5707759Z ok (3.911s)
2022-11-23T02:31:58.5708253Z   test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65970
2022-11-23T02:31:58.5708841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65971
2022-11-23T02:31:58.5709461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5709899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5710472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5710938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5711516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5711944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5712519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5712982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5713419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5714009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5714682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5715374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5715876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5716345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5716704Z dist init r=0, world=2
2022-11-23T02:31:58.5716942Z dist init r=1, world=2
2022-11-23T02:31:58.5717187Z ok (4.312s)
2022-11-23T02:31:58.5717708Z   test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66053
2022-11-23T02:31:58.5718316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66054
2022-11-23T02:31:58.5718913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5719368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5719946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5720412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5721073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5721516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5722088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5722541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5722997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5723496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5724155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5724831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5725356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5725824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5726187Z dist init r=1, world=2
2022-11-23T02:31:58.5726420Z dist init r=0, world=2
2022-11-23T02:31:58.5726655Z ok (4.413s)
2022-11-23T02:31:58.5727149Z   test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_None (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66136
2022-11-23T02:31:58.5727714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66137
2022-11-23T02:31:58.5728331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5728787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5729344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5729820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5730396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5730913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5731480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5731940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5732392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5732890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5733529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5734222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5734744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5735200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5735558Z dist init r=0, world=2
2022-11-23T02:31:58.5735809Z dist init r=1, world=2
2022-11-23T02:31:58.5736049Z ok (4.412s)
2022-11-23T02:31:58.5736545Z   test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66219
2022-11-23T02:31:58.5737149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66220
2022-11-23T02:31:58.5737765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5738287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5738868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5739339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5739918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5740345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5740914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5741382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5741834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5742315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5742966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5743656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5744156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5744629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5744979Z dist init r=0, world=2
2022-11-23T02:31:58.5745231Z dist init r=1, world=2
2022-11-23T02:31:58.5745454Z ok (4.512s)
2022-11-23T02:31:58.5745913Z   test_fsdp_namedtuple (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66302
2022-11-23T02:31:58.5746427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66303
2022-11-23T02:31:58.5747481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5747945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5748619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5749099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5749671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5750112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5750688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5751132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5751592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5752086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5752749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5753421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5753941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5754412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5754770Z dist init r=0, world=2
2022-11-23T02:31:58.5755004Z dist init r=1, world=2
2022-11-23T02:31:58.5755247Z ok (3.911s)
2022-11-23T02:31:58.5755679Z   test_fsdp_not_all_outputs_used_in_loss (__main__.TestFSDPMisc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66381
2022-11-23T02:31:58.5756283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66382
2022-11-23T02:31:58.5756899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5757354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5757913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5758379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5758959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5759399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5759950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5760421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5760874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5761374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5762019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5762703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5763229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5763686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5764581Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:31:58.5765175Z   warnings.warn(message, UserWarning)
2022-11-23T02:31:58.5766046Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:31:58.5766624Z   warnings.warn(message, UserWarning)
2022-11-23T02:31:58.5766889Z dist init r=1, world=2
2022-11-23T02:31:58.5767143Z dist init r=0, world=2
2022-11-23T02:31:58.5767384Z ok (4.515s)
2022-11-23T02:31:58.5767673Z   test_fsdp_same_model_across_ranks (__main__.TestFSDPMisc)
2022-11-23T02:31:58.5768189Z FSDP broadcasts model from rank 0 to ensure it starts off with the same ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66464
2022-11-23T02:31:58.5768734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66465
2022-11-23T02:31:58.5769338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5769791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5770367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5770833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5771395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5771843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5772414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5772953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5773390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5773882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5774548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5775219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5775739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5776213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5776569Z dist init r=0, world=2
2022-11-23T02:31:58.5776803Z dist init r=1, world=2
2022-11-23T02:31:58.5777045Z ok (3.911s)
2022-11-23T02:31:58.5777361Z   test_module_device_mismatches_device_id (__main__.TestFSDPMisc)
2022-11-23T02:31:58.5777862Z Tests that specifying a ``device_id`` argument to FSDP for a GPU ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66543
2022-11-23T02:31:58.5778394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66544
2022-11-23T02:31:58.5779009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5779461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5780020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5780483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5781055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5781487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5782061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5782520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5783046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5783532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5784192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5784884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5785405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5785866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5786222Z dist init r=0, world=2
2022-11-23T02:31:58.5786470Z dist init r=1, world=2
2022-11-23T02:31:58.5786693Z ok (3.910s)
2022-11-23T02:31:58.5787283Z   test_multi_device_not_supported (__main__.TestFSDPMisc)
2022-11-23T02:31:58.5787937Z Tests that wrapping a multi-device module (i.e. with submodules on ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66622
2022-11-23T02:31:58.5788460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66623
2022-11-23T02:31:58.5789073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5789529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5790103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5790653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5791237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5791680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5792240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5792707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5793164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5793661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5794303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5794998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5795521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5795994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5796334Z dist init r=0, world=2
2022-11-23T02:31:58.5796593Z dist init r=1, world=2
2022-11-23T02:31:58.5796834Z ok (4.010s)
2022-11-23T02:31:58.5797085Z   test_no_params (__main__.TestFSDPMisc)
2022-11-23T02:31:58.5797560Z Test that device_id and cpu init work if module has no params ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66701
2022-11-23T02:31:58.5798087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66702
2022-11-23T02:31:58.5798686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5799138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5799715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5800179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5800820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:31:58.5801283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:31:58.5801864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:31:58.5802324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:31:58.5802759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:31:58.5803260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:31:58.5803917Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5804591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:31:58.5805117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:31:58.5805593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:31:58.5805951Z dist init r=1, world=2
2022-11-23T02:31:58.5806180Z dist init r=0, world=2
2022-11-23T02:31:58.5806426Z ok (3.911s)
2022-11-23T02:31:58.5806576Z 
2022-11-23T02:31:58.5806848Z ----------------------------------------------------------------------
2022-11-23T02:31:58.5807162Z Ran 16 tests in 67.401s
2022-11-23T02:31:58.5807327Z 
2022-11-23T02:31:58.5807500Z OK
2022-11-23T02:31:58.5807640Z 
2022-11-23T02:31:58.5807768Z Generating XML reports...
2022-11-23T02:31:58.5808349Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20221123023050.xml
2022-11-23T02:31:58.5808664Z 
2022-11-23T02:31:58.5809011Z ##[endgroup]
2022-11-23T02:31:58.5809603Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_misc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_misc_rdof6abn)
2022-11-23T02:31:58.5809952Z 
2022-11-23T02:31:58.5810231Z Running distributed/fsdp/test_fsdp_checkpoint ... [2022-11-23 02:31:58.564079]
2022-11-23T02:31:58.5810908Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:31:58.564372]
2022-11-23T02:33:14.7483568Z 
2022-11-23T02:33:14.7484614Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_checkpoint
2022-11-23T02:33:14.7485598Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_gtskaox6)
2022-11-23T02:33:14.7485982Z 
2022-11-23T02:33:14.7486097Z Running tests...
2022-11-23T02:33:14.7486610Z ----------------------------------------------------------------------
2022-11-23T02:33:14.7488096Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint
2022-11-23T02:33:14.7489259Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:33:14.7490057Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66815
2022-11-23T02:33:14.7491035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66816
2022-11-23T02:33:14.7492158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7492795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7493709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7494615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7495686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7496147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7496739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7497198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7497656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7498148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7498819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7499515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7500256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7500949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7501543Z dist init r=1, world=2
2022-11-23T02:33:14.7502100Z dist init r=0, world=2
2022-11-23T02:33:14.7502589Z ok (6.263s)
2022-11-23T02:33:14.7503463Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66898
2022-11-23T02:33:14.7504303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66899
2022-11-23T02:33:14.7504945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7505380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7505966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7506435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7507551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7508019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7508615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7509088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7509550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7510247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7510926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7512053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7512948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7513819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7514461Z dist init r=0, world=2
2022-11-23T02:33:14.7514924Z dist init r=1, world=2
2022-11-23T02:33:14.7515411Z ok (4.614s)
2022-11-23T02:33:14.7516379Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66981
2022-11-23T02:33:14.7517163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66982
2022-11-23T02:33:14.7517811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7518244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7518818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7519284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7519844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7520299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7520872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7521335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7521774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7522272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7523064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7523756Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7524261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7524831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7525193Z dist init r=0, world=2
2022-11-23T02:33:14.7525427Z dist init r=1, world=2
2022-11-23T02:33:14.7525670Z ok (4.515s)
2022-11-23T02:33:14.7526239Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67064
2022-11-23T02:33:14.7526889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67065
2022-11-23T02:33:14.7527485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7527931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7528507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7528969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7529548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7529993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7530570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7531019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7531472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7531966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7532623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7533298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7533817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7534288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7534685Z dist init r=0, world=2
2022-11-23T02:33:14.7534949Z dist init r=1, world=2
2022-11-23T02:33:14.7535189Z ok (4.513s)
2022-11-23T02:33:14.7535734Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67147
2022-11-23T02:33:14.7536387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67148
2022-11-23T02:33:14.7537008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7537462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7538019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7538488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7539071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7539513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7540064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7540526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7540982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7541544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7542183Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7542872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7543391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7543844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7544249Z dist init r=0, world=2
2022-11-23T02:33:14.7544500Z dist init r=1, world=2
2022-11-23T02:33:14.7544725Z ok (4.513s)
2022-11-23T02:33:14.7545288Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67230
2022-11-23T02:33:14.7545942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67231
2022-11-23T02:33:14.7546561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7547471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7548075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7548545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7549128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7549552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7550120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7550586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7551038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7551513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7552273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7552986Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7553489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7553963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7554315Z dist init r=0, world=2
2022-11-23T02:33:14.7554568Z dist init r=1, world=2
2022-11-23T02:33:14.7554794Z ok (4.513s)
2022-11-23T02:33:14.7555356Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67313
2022-11-23T02:33:14.7556005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67314
2022-11-23T02:33:14.7556599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7557049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7557627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7558098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7558660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7559193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7559772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7560241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7560682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7561174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7561835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7562503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7563024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7563505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7563861Z dist init r=1, world=2
2022-11-23T02:33:14.7564091Z dist init r=0, world=2
2022-11-23T02:33:14.7564331Z ok (4.513s)
2022-11-23T02:33:14.7564899Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67396
2022-11-23T02:33:14.7565534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67397
2022-11-23T02:33:14.7566150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7566599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7567172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7567627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7568205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7568652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7569262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7569736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7570235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7570731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7571369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7572058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7572580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7573051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7573394Z dist init r=1, world=2
2022-11-23T02:33:14.7573646Z dist init r=0, world=2
2022-11-23T02:33:14.7573884Z ok (4.513s)
2022-11-23T02:33:14.7574433Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67479
2022-11-23T02:33:14.7575081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67480
2022-11-23T02:33:14.7575693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7576219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7576778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7577242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7577821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7578260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7578813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7579273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7579723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7580206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7580863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7581559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7582088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7582540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7582890Z dist init r=0, world=2
2022-11-23T02:33:14.7583142Z dist init r=1, world=2
2022-11-23T02:33:14.7583365Z ok (4.514s)
2022-11-23T02:33:14.7583926Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67562
2022-11-23T02:33:14.7584578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67563
2022-11-23T02:33:14.7585184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7585618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7586248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7586728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7587836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7588280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7588854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7589324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7589759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7590259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7590917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7591606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7592109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7592584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7592939Z dist init r=1, world=2
2022-11-23T02:33:14.7593171Z dist init r=0, world=2
2022-11-23T02:33:14.7593521Z ok (4.512s)
2022-11-23T02:33:14.7594087Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67645
2022-11-23T02:33:14.7594744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67646
2022-11-23T02:33:14.7595338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7595787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7596363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7596830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7597385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7597831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7598398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7598842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7599296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7599790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7600446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7601117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7601635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7602113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7602453Z dist init r=0, world=2
2022-11-23T02:33:14.7602708Z dist init r=1, world=2
2022-11-23T02:33:14.7602949Z ok (4.513s)
2022-11-23T02:33:14.7603583Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67728
2022-11-23T02:33:14.7604229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67729
2022-11-23T02:33:14.7604843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7605295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7605852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7606325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7606911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7607351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7607904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7608371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7608824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7609316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7609956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7610738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7611261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7611717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7612079Z dist init r=1, world=2
2022-11-23T02:33:14.7612333Z dist init r=0, world=2
2022-11-23T02:33:14.7612570Z ok (4.513s)
2022-11-23T02:33:14.7613112Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67811
2022-11-23T02:33:14.7613754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67812
2022-11-23T02:33:14.7614369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7614805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7615383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7615845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7616424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7616850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7617417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7617874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7618323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7618804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7619461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7620230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7620742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7621214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7621566Z dist init r=1, world=2
2022-11-23T02:33:14.7621817Z dist init r=0, world=2
2022-11-23T02:33:14.7622040Z ok (4.413s)
2022-11-23T02:33:14.7622600Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67894
2022-11-23T02:33:14.7623250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67895
2022-11-23T02:33:14.7623846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7624293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7624876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7625342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7625901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7626347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7627412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7628009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7628441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7628937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7629612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7630282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7630803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7631275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7631634Z dist init r=0, world=2
2022-11-23T02:33:14.7631870Z dist init r=1, world=2
2022-11-23T02:33:14.7632113Z ok (4.513s)
2022-11-23T02:33:14.7649273Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67977
2022-11-23T02:33:14.7650015Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67978
2022-11-23T02:33:14.7650663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7651123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7651703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7652159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7652736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7653201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7653772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7654226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7654821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7655345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7655996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7656684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7657207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7657683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7658024Z dist init r=1, world=2
2022-11-23T02:33:14.7658275Z dist init r=0, world=2
2022-11-23T02:33:14.7658519Z ok (4.412s)
2022-11-23T02:33:14.7659067Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68060
2022-11-23T02:33:14.7659717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68061
2022-11-23T02:33:14.7660332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7660787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7661338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7661886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7662472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:33:14.7662913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:33:14.7663464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:33:14.7663925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:33:14.7664379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:33:14.7664863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:33:14.7665526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7666220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:33:14.7666741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:33:14.7667742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:33:14.7668111Z dist init r=0, world=2
2022-11-23T02:33:14.7668364Z dist init r=1, world=2
2022-11-23T02:33:14.7668583Z ok (4.512s)
2022-11-23T02:33:14.7668731Z 
2022-11-23T02:33:14.7669018Z ----------------------------------------------------------------------
2022-11-23T02:33:14.7669353Z Ran 16 tests in 73.861s
2022-11-23T02:33:14.7669515Z 
2022-11-23T02:33:14.7669611Z OK
2022-11-23T02:33:14.7669724Z 
2022-11-23T02:33:14.7669851Z Generating XML reports...
2022-11-23T02:33:14.7670466Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20221123023200.xml
2022-11-23T02:33:14.7670834Z 
2022-11-23T02:33:14.7671184Z ##[endgroup]
2022-11-23T02:33:14.7671782Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_gtskaox6)
2022-11-23T02:33:14.7672153Z 
2022-11-23T02:33:14.7672405Z Running distributed/test_c10d_pypg ... [2022-11-23 02:33:14.748621]
2022-11-23T02:33:14.7673164Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:33:14.748886]
2022-11-23T02:35:40.0644987Z 
2022-11-23T02:35:40.0650064Z Expand the folded group to see the log file of distributed/test_c10d_pypg
2022-11-23T02:35:40.0651203Z ##[group]PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_pypg_a2ad6dyr)
2022-11-23T02:35:40.0651560Z 
2022-11-23T02:35:40.0651675Z Running tests...
2022-11-23T02:35:40.0652187Z ----------------------------------------------------------------------
2022-11-23T02:35:40.0652761Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_pypg
2022-11-23T02:35:40.0653230Z   test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0654561Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:35:40.0655636Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68178
2022-11-23T02:35:40.0656931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0657470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0658047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0658519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0659217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0659722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8rrx4m32
2022-11-23T02:35:40.0660244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8rrx4m32/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0660647Z ok (5.756s)
2022-11-23T02:35:40.0661016Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0661571Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68215
2022-11-23T02:35:40.0662295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0662749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0663335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0663790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0664235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0664843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo8ep643
2022-11-23T02:35:40.0668917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo8ep643/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0669732Z ok (4.110s)
2022-11-23T02:35:40.0670497Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0671077Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68252
2022-11-23T02:35:40.0671807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0672435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0673710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0674649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0675676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0676580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjyorokue
2022-11-23T02:35:40.0677544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjyorokue/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0678454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0679298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0681522Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:35:40.0682721Z   warnings.warn(
2022-11-23T02:35:40.0683111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0683597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0684059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0684534Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0684876Z ok (4.210s)
2022-11-23T02:35:40.0685215Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0685763Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68289
2022-11-23T02:35:40.0686607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0687061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0687621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0688089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0688526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0689028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwl8lvncb
2022-11-23T02:35:40.0689549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwl8lvncb/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0690065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0690552Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0691703Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:35:40.0692427Z   warnings.warn(
2022-11-23T02:35:40.0692801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0693280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0693733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0694212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0694556Z ok (4.210s)
2022-11-23T02:35:40.0694934Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0695692Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68326
2022-11-23T02:35:40.0696399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0696847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0697403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0697868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0698310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0698819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpko9jxm2v
2022-11-23T02:35:40.0699338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpko9jxm2v/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0699849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0700332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0700685Z ok (4.110s)
2022-11-23T02:35:40.0701043Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0701751Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68363
2022-11-23T02:35:40.0702437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0702936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0703518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0703986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0704430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0704918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcafo7mpe
2022-11-23T02:35:40.0705457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcafo7mpe/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0705971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0706433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0707366Z ok (4.210s)
2022-11-23T02:35:40.0707795Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0708534Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68400
2022-11-23T02:35:40.0709220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0709671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0710243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0710710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0711129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0711628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2qmeiic
2022-11-23T02:35:40.0712163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2qmeiic/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0712664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0713788Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:35:40.0714792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0715142Z ok (4.310s)
2022-11-23T02:35:40.0715503Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0716208Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68437
2022-11-23T02:35:40.0716912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0717370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0717928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0718399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0718840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0719349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63vdbj1_
2022-11-23T02:35:40.0719868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63vdbj1_/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0720339Z ok (4.210s)
2022-11-23T02:35:40.0720699Z   test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0721255Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68474
2022-11-23T02:35:40.0721966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0722421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0722997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0723444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0723887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0724388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptn4m17_g
2022-11-23T02:35:40.0724933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptn4m17_g/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0725431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0725919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0726272Z ok (4.210s)
2022-11-23T02:35:40.0726631Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0727215Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68511
2022-11-23T02:35:40.0727918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0728370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0728934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0729407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0729847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0730408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprb1rn8jp
2022-11-23T02:35:40.0730945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprb1rn8jp/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0731998Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:35:40.0733678Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:35:40.0734395Z   warnings.warn(
2022-11-23T02:35:40.0734776Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0735246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0735596Z ok (4.210s)
2022-11-23T02:35:40.0735967Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0736603Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68548
2022-11-23T02:35:40.0737313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0737770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0738349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0738800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0739242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0739745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8arp0rdi
2022-11-23T02:35:40.0740290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8arp0rdi/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0741477Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:35:40.0742315Z   warnings.warn(
2022-11-23T02:35:40.0742696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0743181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0743514Z ok (4.111s)
2022-11-23T02:35:40.0743885Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0744441Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68585
2022-11-23T02:35:40.0745108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0745559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0746192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0746671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0747660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0748169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0cbntnz
2022-11-23T02:35:40.0748711Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0cbntnz/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0749228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0749698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0750050Z ok (4.110s)
2022-11-23T02:35:40.0750423Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2022-11-23T02:35:40.0750964Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68622
2022-11-23T02:35:40.0751660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0752113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0752690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0753140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0753583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0754203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1nbwrqr_
2022-11-23T02:35:40.0754731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1nbwrqr_/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0755249Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0755739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0756221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0756678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0757027Z ok (4.210s)
2022-11-23T02:35:40.0757477Z   test_ddp_invoke_work_object (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68659
2022-11-23T02:35:40.0758166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0758618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0759192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0759662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0760084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0760585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgrw4wk14
2022-11-23T02:35:40.0761124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgrw4wk14/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0761488Z ok (2.206s)
2022-11-23T02:35:40.0761925Z   test_ddp_with_pypg (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68695
2022-11-23T02:35:40.0762617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0763067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0763621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0764167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0764621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0765124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpccph5rox
2022-11-23T02:35:40.0765646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpccph5rox/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0766167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0766521Z ok (2.242s)
2022-11-23T02:35:40.0766954Z   test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68731
2022-11-23T02:35:40.0767674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0768125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0768705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0769160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0769601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0770101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl841nzah
2022-11-23T02:35:40.0770624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl841nzah/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0771206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0771559Z ok (2.206s)
2022-11-23T02:35:40.0772007Z   test_invalid_powerSGD_state (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68767
2022-11-23T02:35:40.0772692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0773145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0773718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0774191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0774608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0775400Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:35:40.0776485Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:35:40.0777554Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:35:40.0778626Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:35:40.0779752Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:35:40.0780821Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:35:40.0781429Z ok (2.206s)
2022-11-23T02:35:40.0781877Z   test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68801
2022-11-23T02:35:40.0782589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0783041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0783600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0784069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0784511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0784994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeyyqcgyy
2022-11-23T02:35:40.0785601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeyyqcgyy/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0786121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0786474Z ok (3.609s)
2022-11-23T02:35:40.0787530Z   test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68838
2022-11-23T02:35:40.0788271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0788720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0789275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0789747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0790185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0790695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmdc63hw
2022-11-23T02:35:40.0791218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmdc63hw/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0791735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0792088Z ok (3.710s)
2022-11-23T02:35:40.0792410Z   test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0793111Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68875
2022-11-23T02:35:40.0793810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0794261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0794819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0795286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0795730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0796334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9h_2nl1l
2022-11-23T02:35:40.0796869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9h_2nl1l/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0797250Z ok (4.111s)
2022-11-23T02:35:40.0797607Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0798159Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68912
2022-11-23T02:35:40.0798871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0799325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0799898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0800352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0800797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0801300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxokmg97h
2022-11-23T02:35:40.0801819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxokmg97h/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0802201Z ok (4.110s)
2022-11-23T02:35:40.0802558Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0803112Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68949
2022-11-23T02:35:40.0803875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0804326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0804900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0805370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0805789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0806293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yxzb694
2022-11-23T02:35:40.0806831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yxzb694/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0807327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0807817Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0808986Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:35:40.0809713Z   warnings.warn(
2022-11-23T02:35:40.0810069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0810554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0811031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0811512Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0811841Z ok (4.310s)
2022-11-23T02:35:40.0812194Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0812747Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68986
2022-11-23T02:35:40.0813472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0813934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0814515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0814989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0815410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0815907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_d6baus
2022-11-23T02:35:40.0816451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_d6baus/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0816969Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0817440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0818603Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:35:40.0819332Z   warnings.warn(
2022-11-23T02:35:40.0819710Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0820235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0820711Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0821190Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0821517Z ok (4.310s)
2022-11-23T02:35:40.0821899Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0822614Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69023
2022-11-23T02:35:40.0823305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0823738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0824312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0824786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0825208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0825714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__qny6ol
2022-11-23T02:35:40.0826256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__qny6ol/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0827220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0827788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0828139Z ok (4.210s)
2022-11-23T02:35:40.0828521Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0829240Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69060
2022-11-23T02:35:40.0829924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0830375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0831047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0831513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0831953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0832457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy350p848
2022-11-23T02:35:40.0833004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy350p848/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0833500Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0833994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0834345Z ok (4.111s)
2022-11-23T02:35:40.0834683Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0835407Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69097
2022-11-23T02:35:40.0836118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0836569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0837125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0837596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0838033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0838620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphs3r884i
2022-11-23T02:35:40.0839143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphs3r884i/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0839663Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0840692Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:35:40.0841726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0842061Z ok (4.310s)
2022-11-23T02:35:40.0842418Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0843139Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69134
2022-11-23T02:35:40.0843843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0844278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0844853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0845322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0845744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0846248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpz_l02k
2022-11-23T02:35:40.0846785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpz_l02k/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0847162Z ok (4.210s)
2022-11-23T02:35:40.0847557Z   test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0848137Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69171
2022-11-23T02:35:40.0848846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0849296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0849852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0850325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0850763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0851248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7pfpygf
2022-11-23T02:35:40.0851797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7pfpygf/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0852316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0852800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0853134Z ok (4.110s)
2022-11-23T02:35:40.0853504Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0854089Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69208
2022-11-23T02:35:40.0854838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0855292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0855871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0856343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0856763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0857266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1mscxzc
2022-11-23T02:35:40.0857809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1mscxzc/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0858869Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:35:40.0860534Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:35:40.0861237Z   warnings.warn(
2022-11-23T02:35:40.0861617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0862102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0862458Z ok (4.210s)
2022-11-23T02:35:40.0862805Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0863439Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69245
2022-11-23T02:35:40.0864156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0864590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0865166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0865636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0866076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0866568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzh773n2j
2022-11-23T02:35:40.0867775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzh773n2j/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0868995Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1862: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-11-23T02:35:40.0869723Z   warnings.warn(
2022-11-23T02:35:40.0870081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0870570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0870924Z ok (4.210s)
2022-11-23T02:35:40.0871413Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0871945Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69282
2022-11-23T02:35:40.0872632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0873087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0873644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0874118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0874560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0875067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6bqhew0
2022-11-23T02:35:40.0875595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6bqhew0/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0876114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0876598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0876933Z ok (4.212s)
2022-11-23T02:35:40.0877310Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2022-11-23T02:35:40.0877858Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69319
2022-11-23T02:35:40.0878538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0878975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0879547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0880022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0880463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0880945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2h7oc_i9
2022-11-23T02:35:40.0881557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2h7oc_i9/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0882087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0882557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0883037Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0883514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0883863Z ok (4.212s)
2022-11-23T02:35:40.0884297Z   test_ddp_invoke_work_object (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69356
2022-11-23T02:35:40.0885000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0885454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0886011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0886483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0886925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0887427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphntdgub1
2022-11-23T02:35:40.0887951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphntdgub1/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0888401Z ok (2.207s)
2022-11-23T02:35:40.0888839Z   test_ddp_with_pypg (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69392
2022-11-23T02:35:40.0889512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0889965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0890538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0891008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0891431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0891932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_nl13lj
2022-11-23T02:35:40.0892468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_nl13lj/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0892988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0893324Z ok (2.206s)
2022-11-23T02:35:40.0893776Z   test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69428
2022-11-23T02:35:40.0894485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0894915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0895487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0895957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0896398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0896886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ha28ian
2022-11-23T02:35:40.0897421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ha28ian/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0897935Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0898271Z ok (2.206s)
2022-11-23T02:35:40.0898795Z   test_invalid_powerSGD_state (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69464
2022-11-23T02:35:40.0899508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0899958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0900506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0900972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0901415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0902218Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:35:40.0903276Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:35:40.0904343Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:35:40.0905467Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:35:40.0906530Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:35:40.0907788Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-11-23T02:35:40.0908430Z ok (2.206s)
2022-11-23T02:35:40.0908891Z   test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69498
2022-11-23T02:35:40.0909590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0910042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0910618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0911089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0911517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0912018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm_afkw5m
2022-11-23T02:35:40.0912556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm_afkw5m/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0913143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0913516Z ok (3.709s)
2022-11-23T02:35:40.0913973Z   test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69535
2022-11-23T02:35:40.0914682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:35:40.0915112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:35:40.0915689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:35:40.0916163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:35:40.0916580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:35:40.0917086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4uh511gk
2022-11-23T02:35:40.0917623Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4uh511gk/_remote_module_non_scriptable.py
2022-11-23T02:35:40.0918138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:35:40.0918474Z ok (3.710s)
2022-11-23T02:35:40.0918622Z 
2022-11-23T02:35:40.0918900Z ----------------------------------------------------------------------
2022-11-23T02:35:40.0919235Z Ran 38 tests in 143.039s
2022-11-23T02:35:40.0919404Z 
2022-11-23T02:35:40.0919480Z OK
2022-11-23T02:35:40.0919616Z 
2022-11-23T02:35:40.0919971Z Generating XML reports...
2022-11-23T02:35:40.0920691Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20221123023316.xml
2022-11-23T02:35:40.0921476Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20221123023316.xml
2022-11-23T02:35:40.0921836Z 
2022-11-23T02:35:40.0922237Z ##[endgroup]
2022-11-23T02:35:40.0922818Z FINISHED PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_pypg_a2ad6dyr)
2022-11-23T02:35:40.0923154Z 
2022-11-23T02:35:40.0923446Z Running distributed/fsdp/test_fsdp_use_orig_params ... [2022-11-23 02:35:40.064885]
2022-11-23T02:35:40.0924153Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:35:40.065154]
2022-11-23T02:38:42.6102658Z 
2022-11-23T02:38:42.6103612Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_use_orig_params
2022-11-23T02:38:42.6105059Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_47te5edq)
2022-11-23T02:38:42.6105685Z 
2022-11-23T02:38:42.6105811Z Running tests...
2022-11-23T02:38:42.6106322Z ----------------------------------------------------------------------
2022-11-23T02:38:42.6107225Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params
2022-11-23T02:38:42.6107733Z   test_named_parameters_in_forward (__main__.TestFSDPUseOrigParamsFQNs)
2022-11-23T02:38:42.6108227Z Tests that calling ``named_parameters()`` during forward returns FQNs ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T02:38:42.6109529Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69607
2022-11-23T02:38:42.6109989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69608
2022-11-23T02:38:42.6110647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6111103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6111663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6114044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6114782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6115261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6116256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6117063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6117864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6118837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6120184Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6121440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6122425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6123321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6123984Z dist init r=1, world=2
2022-11-23T02:38:42.6124406Z dist init r=0, world=2
2022-11-23T02:38:42.6124872Z ok (6.055s)
2022-11-23T02:38:42.6125519Z   test_param_and_buffer_names (__main__.TestFSDPUseOrigParamsFQNs)
2022-11-23T02:38:42.6126471Z Tests that, for ``use_orig_params=True``, the parameter and buffer ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69686
2022-11-23T02:38:42.6127676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69687
2022-11-23T02:38:42.6128833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6129635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6130703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6131613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6132754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6133187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6133763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6134234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6134978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6135467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6136130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6136820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6137347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6137805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6139196Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:38:42.6139989Z   warnings.warn(
2022-11-23T02:38:42.6141153Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:38:42.6141927Z   warnings.warn(
2022-11-23T02:38:42.6142156Z dist init r=0, world=2
2022-11-23T02:38:42.6142404Z dist init r=1, world=2
2022-11-23T02:38:42.6142639Z ok (4.011s)
2022-11-23T02:38:42.6143059Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:38:42.6143697Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69765
2022-11-23T02:38:42.6144237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69766
2022-11-23T02:38:42.6144854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6145291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6145863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6146397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6147488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6147932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6148524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6148988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6149422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6149919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6150576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6151271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6151779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6152250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6152732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6153214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6153560Z dist init r=0, world=2
2022-11-23T02:38:42.6153811Z dist init r=1, world=2
2022-11-23T02:38:42.6154053Z ok (5.413s)
2022-11-23T02:38:42.6154468Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:38:42.6155118Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69848
2022-11-23T02:38:42.6155660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69849
2022-11-23T02:38:42.6156277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6156712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6157396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6157881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6158446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6158898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6159465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6159933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6160373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6160870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6161527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6162220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6162725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6163197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6163673Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6164225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6164585Z dist init r=0, world=2
2022-11-23T02:38:42.6164835Z dist init r=1, world=2
2022-11-23T02:38:42.6165055Z ok (5.414s)
2022-11-23T02:38:42.6165499Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:38:42.6166138Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69931
2022-11-23T02:38:42.6166682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69932
2022-11-23T02:38:42.6167281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6167728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6168295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6168745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6169300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6169774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6170359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6170806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6171258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6171753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6172410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6173084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6173603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6174126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6174610Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6175073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6175435Z dist init r=0, world=2
2022-11-23T02:38:42.6175683Z dist init r=1, world=2
2022-11-23T02:38:42.6175904Z ok (5.414s)
2022-11-23T02:38:42.6176325Z   test_diff_hyperparams_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:38:42.6176952Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70014
2022-11-23T02:38:42.6177479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70015
2022-11-23T02:38:42.6178093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6178544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6179112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6179562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6180136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6180574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6181147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6181665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6182113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6182610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6183254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6183942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6184462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6184933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6185397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6185888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6186368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6187234Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6187731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6188207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6188676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6189128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6189600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6190073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6190541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6190994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6191555Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6192054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6192498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6192967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6193431Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6193895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6194356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6194821Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6195284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6195736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6196207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6196669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6197139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6197585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6198048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6198606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6199050Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6199516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6199981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6200444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6200893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6201355Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6201814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6202282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6202730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6203189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6203656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6204108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6204571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6205028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6205490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6205938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6206406Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6206865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6207315Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6207829Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6209122Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:38:42.6209901Z   warnings.warn(
2022-11-23T02:38:42.6211052Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:38:42.6211819Z   warnings.warn(
2022-11-23T02:38:42.6212171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6212650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6213127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6213581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6214048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6214579Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6215029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6215504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6215972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6216441Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6216890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6217356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6217819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6218293Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6218742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6219206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6219670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6220118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6220579Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6221040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6221503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6221954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6222416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6222877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6223325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6223852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6224324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6224787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6225239Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6225707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6226176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6226632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6227590Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6228062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6228533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6228986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6229454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6229915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6230382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6230833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6231400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6231865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6232323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6232789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6233259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6233729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6234183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6234649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6235007Z dist init r=0, world=2
2022-11-23T02:38:42.6235237Z dist init r=1, world=2
2022-11-23T02:38:42.6235474Z ok (33.155s)
2022-11-23T02:38:42.6235895Z   test_diff_hyperparams_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:38:42.6236504Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70097
2022-11-23T02:38:42.6237042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70098
2022-11-23T02:38:42.6237671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6238122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6238723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6239186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6239773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6240214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6240840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6241311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6241761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6242237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6242902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6243587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6244111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6244559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6245030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6245514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6245993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6246451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6247455Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6248763Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6249993Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6251205Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6252438Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6253634Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6254854Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6256064Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6257320Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6258536Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6259254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6259742Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6260203Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6260678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6261149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6261599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6262069Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6262538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6263062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6263518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6263982Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6264451Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6264903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6265377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6265855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6266328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6267097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6267669Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6268137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6268588Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6269066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6269530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6269990Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6270445Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6271453Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6272781Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6274025Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6275239Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6276468Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6277679Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6278879Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6280168Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6281369Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6282577Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6283295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6283778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6284238Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6284714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6285184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6285634Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6286109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6286578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6287053Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6287508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6288550Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6289790Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6291003Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6292229Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6293440Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6294656Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6295917Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6297129Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6298341Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6299554Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6300278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6300760Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6301219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6301692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6302683Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6303946Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6305173Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6306367Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6308004Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6309220Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6310539Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6311752Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6312951Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6314167Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6314888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6315357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6315835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6316308Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6316761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6317231Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6318571Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:38:42.6319356Z   warnings.warn(
2022-11-23T02:38:42.6320503Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:38:42.6321275Z   warnings.warn(
2022-11-23T02:38:42.6321628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6322104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6322585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6323041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6323506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6323977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6324441Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6324896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6325428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6325897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6326352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6326830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6327301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6327771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6328223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6328691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6329160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6329616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6330624Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6331867Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6333091Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6334373Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6335607Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6336821Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6338051Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6339293Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6340500Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6341771Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6342497Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6342981Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6343439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6343911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6344381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6344853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6345305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6345768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6346240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6346684Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6347640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6348112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6348576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6349030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6349502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6349966Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6350532Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6351002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6351470Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6351938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6352390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6352860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6353333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6353800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6354800Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6356046Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6357271Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6358577Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6359793Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6361006Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6362230Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6363451Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6364653Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6365922Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:38:42.6366651Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6367131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6367615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6368071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6368550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6369016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6369354Z dist init r=0, world=2
2022-11-23T02:38:42.6369602Z dist init r=1, world=2
2022-11-23T02:38:42.6369837Z ok (33.657s)
2022-11-23T02:38:42.6370256Z   test_diff_hyperparams_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:38:42.6370887Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70180
2022-11-23T02:38:42.6371426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70181
2022-11-23T02:38:42.6372044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6372479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6373118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6373583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6374165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6374589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6375161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6375621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6376056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6376553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6377215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6377906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6378413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6378884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6379359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6379839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6380303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6380778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6381249Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6381706Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6382177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6382698Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6383176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6383629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6384095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6384562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6385013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6385484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6385944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6386411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6387099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6387580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6388046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6388493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6388961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6389425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6389986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6390439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6390904Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6391371Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6391835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6392283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6392750Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6393213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6393662Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6394127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6394589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6395055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6395502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6395964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6396429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6396874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6397334Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6397803Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6398266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6398714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6399241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6399713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6400157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6400623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6401082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6401550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6402825Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:38:42.6403606Z   warnings.warn(
2022-11-23T02:38:42.6404757Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:608: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2022-11-23T02:38:42.6405583Z   warnings.warn(
2022-11-23T02:38:42.6405954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6406416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6406900Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6407375Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6407847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6408303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6408770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6409235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6409694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6410165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6410632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6411157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6411611Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6412072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6412539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6412988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6413454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6413921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6414385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6414833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6415354Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6415824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6416271Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6416733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6417196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6417660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6418109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6418573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6419031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6419480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6419948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6420416Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6420881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6421333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6421798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6422332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6422804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6423253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6423718Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6424183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6424630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6425092Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6425556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6426021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6426472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6427099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6427573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6428020Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6428373Z dist init r=1, world=2
2022-11-23T02:38:42.6428621Z dist init r=0, world=2
2022-11-23T02:38:42.6428840Z ok (32.955s)
2022-11-23T02:38:42.6429215Z   test_diff_trainability (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:38:42.6429804Z Tests FSDP parity with DDP when using multiple parameter groups and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70263
2022-11-23T02:38:42.6430340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70264
2022-11-23T02:38:42.6430952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6431402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6432057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6432538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6433099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6433539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6434107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6434553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6435009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6435501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6436162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6436835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6437351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6437821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6438174Z dist init r=0, world=2
2022-11-23T02:38:42.6438444Z dist init r=1, world=2
2022-11-23T02:38:42.6438685Z ok (8.218s)
2022-11-23T02:38:42.6439067Z   test_multiple_optimizers (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2022-11-23T02:38:42.6439698Z Tests using two optimizers where only one sets gradients to ``None``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70346
2022-11-23T02:38:42.6440224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70347
2022-11-23T02:38:42.6440846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6441279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6441849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6442318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6442896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6443322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6443896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6444358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6444817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6445291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6445945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6446629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6447129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6447596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6448663Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:38:42.6450183Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-11-23T02:38:42.6451041Z dist init r=0, world=2
2022-11-23T02:38:42.6451291Z dist init r=1, world=2
2022-11-23T02:38:42.6451527Z ok (5.112s)
2022-11-23T02:38:42.6451875Z   test_access_params_after_forward (__main__.TestFSDPUseOrigParamsParamAccess)
2022-11-23T02:38:42.6452440Z Tests that accessing the original parameters after the forward but ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70429
2022-11-23T02:38:42.6452974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70430
2022-11-23T02:38:42.6453572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6454019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6454595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6455122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6455686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6456134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6456707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6457170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6457606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6458104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6458759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6459426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6459947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6460418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6460897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6461362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6461841Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6462318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6462770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6463246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6463601Z dist init r=0, world=2
2022-11-23T02:38:42.6463848Z dist init r=1, world=2
2022-11-23T02:38:42.6464069Z ok (4.512s)
2022-11-23T02:38:42.6464453Z   test_multiple_forward_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard)
2022-11-23T02:38:42.6465087Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70512
2022-11-23T02:38:42.6465601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70513
2022-11-23T02:38:42.6466216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6466663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6467480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6467940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6468518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6468955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6469508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6469974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6470421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6470911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6471547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6472333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6472849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6473326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6473664Z dist init r=0, world=2
2022-11-23T02:38:42.6473913Z dist init r=1, world=2
2022-11-23T02:38:42.6474152Z ok (5.713s)
2022-11-23T02:38:42.6474516Z   test_multiple_forward_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard)
2022-11-23T02:38:42.6475093Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70595
2022-11-23T02:38:42.6475614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70596
2022-11-23T02:38:42.6476228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6476667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6477238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6477709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6478270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6478712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6479281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6479744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6480179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6480682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6481341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6482070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6482597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6483066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6483417Z dist init r=1, world=2
2022-11-23T02:38:42.6483649Z dist init r=0, world=2
2022-11-23T02:38:42.6483889Z ok (5.814s)
2022-11-23T02:38:42.6484291Z   test_summon_between_two_forwards_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard)
2022-11-23T02:38:42.6484861Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70678
2022-11-23T02:38:42.6485395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70679
2022-11-23T02:38:42.6486011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6486462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6487019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6487486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6488058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6488503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6489050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6489577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6507685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6508273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6508996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6509689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6510214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6510672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6511027Z dist init r=0, world=2
2022-11-23T02:38:42.6511280Z dist init r=1, world=2
2022-11-23T02:38:42.6511501Z ok (6.115s)
2022-11-23T02:38:42.6511905Z   test_summon_between_two_forwards_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard)
2022-11-23T02:38:42.6512500Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70761
2022-11-23T02:38:42.6513026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70762
2022-11-23T02:38:42.6513630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6514087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6514663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6515138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6515709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6516159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6516861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6517328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6517786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6518289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6518951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6519623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6520151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6520624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6520964Z dist init r=0, world=2
2022-11-23T02:38:42.6521218Z dist init r=1, world=2
2022-11-23T02:38:42.6521462Z ok (6.214s)
2022-11-23T02:38:42.6521780Z   test_grad_writeback (__main__.TestFSDPUseOrigParamsWriteback)
2022-11-23T02:38:42.6522458Z Tests that changes to the original parameters' gradients are written ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70844
2022-11-23T02:38:42.6522999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70845
2022-11-23T02:38:42.6523609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6524041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6524702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6525174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6525757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6526181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6526749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6527213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6527652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6528156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6528818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6529510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6530019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6530494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6530975Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6531461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6531925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6532400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6532883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6533341Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6533824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6534351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6534831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6535289Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6535761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6536233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-11-23T02:38:42.6536571Z dist init r=0, world=2
2022-11-23T02:38:42.6536822Z dist init r=1, world=2
2022-11-23T02:38:42.6537065Z ok (4.511s)
2022-11-23T02:38:42.6537384Z   test_param_writeback (__main__.TestFSDPUseOrigParamsWriteback)
2022-11-23T02:38:42.6537921Z Tests that changes to the original parameters are written back. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70927
2022-11-23T02:38:42.6538503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70928
2022-11-23T02:38:42.6539127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6539560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6540132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6540598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6541176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6541671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6542247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6542708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6543149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6543651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6544313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6545007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6545512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6545988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6546345Z dist init r=1, world=2
2022-11-23T02:38:42.6546577Z dist init r=0, world=2
2022-11-23T02:38:42.6547018Z ok (4.011s)
2022-11-23T02:38:42.6547488Z   test_writeback_shape_mismatch (__main__.TestFSDPUseOrigParamsWriteback) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71006
2022-11-23T02:38:42.6548057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71007
2022-11-23T02:38:42.6548659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6549103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6549681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6550157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6550723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:38:42.6551165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:38:42.6551812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:38:42.6552271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:38:42.6552722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:38:42.6553222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:38:42.6553880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6554552Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:38:42.6555079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:38:42.6555549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:38:42.6555885Z dist init r=1, world=2
2022-11-23T02:38:42.6556140Z dist init r=0, world=2
2022-11-23T02:38:42.6556381Z ok (3.911s)
2022-11-23T02:38:42.6556532Z 
2022-11-23T02:38:42.6556802Z ----------------------------------------------------------------------
2022-11-23T02:38:42.6557120Z Ran 18 tests in 180.208s
2022-11-23T02:38:42.6557284Z 
2022-11-23T02:38:42.6557378Z OK
2022-11-23T02:38:42.6557511Z 
2022-11-23T02:38:42.6557636Z Generating XML reports...
2022-11-23T02:38:42.6558269Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20221123023541.xml
2022-11-23T02:38:42.6559198Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20221123023541.xml
2022-11-23T02:38:42.6560241Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20221123023541.xml
2022-11-23T02:38:42.6561154Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20221123023541.xml
2022-11-23T02:38:42.6562041Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20221123023541.xml
2022-11-23T02:38:42.6562441Z 
2022-11-23T02:38:42.6562829Z ##[endgroup]
2022-11-23T02:38:42.6563462Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_47te5edq)
2022-11-23T02:38:42.6563833Z 
2022-11-23T02:38:42.6564130Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2022-11-23 02:38:42.611008]
2022-11-23T02:38:42.6564827Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:38:42.611273]
2022-11-23T02:44:01.3107358Z 
2022-11-23T02:44:01.3109551Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_mixed_precision
2022-11-23T02:44:01.3110543Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_pazbq4_p)
2022-11-23T02:44:01.3110933Z 
2022-11-23T02:44:01.3111049Z Running tests...
2022-11-23T02:44:01.3111550Z ----------------------------------------------------------------------
2022-11-23T02:44:01.3114075Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision
2022-11-23T02:44:01.3114742Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71120
2022-11-23T02:44:01.3115315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71121
2022-11-23T02:44:01.3115956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3116658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3118134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3118623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3119193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3119621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3120192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3122815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3123556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3124102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3125065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3125956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3126552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3127132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3127645Z dist init r=0, world=2
2022-11-23T02:44:01.3128084Z dist init r=1, world=2
2022-11-23T02:44:01.3128375Z ok (5.928s)
2022-11-23T02:44:01.3129151Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71204
2022-11-23T02:44:01.3130069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71205
2022-11-23T02:44:01.3130753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3131386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3132061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3132671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3133395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3133900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3134747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3135278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3135930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3136413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3137345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3138193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3138789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3139321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3140526Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3141424Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3142503Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3143081Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3144271Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3145796Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3147874Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3149424Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3151068Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3152557Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3154016Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3155540Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3157096Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3158602Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3159242Z dist init r=1, world=2
2022-11-23T02:44:01.3159496Z dist init r=0, world=2
2022-11-23T02:44:01.3159848Z ok (6.415s)
2022-11-23T02:44:01.3160586Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71288
2022-11-23T02:44:01.3161456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71289
2022-11-23T02:44:01.3162093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3162742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3163343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3163977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3164625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3165133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3165891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3166363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3167022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3167524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3168428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3169381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3169957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3170492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3171565Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3172409Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3173200Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3174059Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3174993Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3176489Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3177737Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3179011Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3180529Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3181776Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3183004Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3184216Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3185427Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3186700Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3187877Z dist init r=1, world=2
2022-11-23T02:44:01.3188112Z dist init r=0, world=2
2022-11-23T02:44:01.3188354Z ok (6.315s)
2022-11-23T02:44:01.3188913Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71372
2022-11-23T02:44:01.3189548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71373
2022-11-23T02:44:01.3190172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3190634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3191209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3191664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3192241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3192681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3193232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3193694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3194144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3194641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3195284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3196066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3196598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3197065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3197931Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3198510Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3199305Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3199876Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3200787Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3202317Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3203679Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3204907Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3206122Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3207349Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3208560Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3209781Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3210979Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3212254Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3212863Z dist init r=1, world=2
2022-11-23T02:44:01.3213112Z dist init r=0, world=2
2022-11-23T02:44:01.3213346Z ok (6.314s)
2022-11-23T02:44:01.3213862Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71456
2022-11-23T02:44:01.3214488Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71457
2022-11-23T02:44:01.3215104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3215536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3216111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3216580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3217155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3217579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3218145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3218668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3219103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3219603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3220265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3220950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3221448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3221916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3222791Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3223378Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3224162Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3224726Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3225634Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3227172Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3228928Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3230312Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3231535Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3232785Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3234005Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3235222Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3236540Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3237750Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3238344Z dist init r=0, world=2
2022-11-23T02:44:01.3238576Z dist init r=1, world=2
2022-11-23T02:44:01.3239300Z ok (6.315s)
2022-11-23T02:44:01.3239873Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71540
2022-11-23T02:44:01.3240529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71541
2022-11-23T02:44:01.3241131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3241583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3242155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3242624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3243180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3243630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3244201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3244648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3245158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3245661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3246318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3246988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3247508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3247985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3248873Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3249440Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3250235Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3250813Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3251726Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3253032Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3254231Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3255456Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3256682Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3257888Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3259101Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3260378Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3261614Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3262832Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3263423Z dist init r=0, world=2
2022-11-23T02:44:01.3263673Z dist init r=1, world=2
2022-11-23T02:44:01.3263909Z ok (6.315s)
2022-11-23T02:44:01.3264426Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71624
2022-11-23T02:44:01.3265048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71625
2022-11-23T02:44:01.3265663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3266113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3266671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3267520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3268110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3268550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3269103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3269561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3270012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3270487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3271144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3271835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3272356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3272809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3273688Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3274269Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3275068Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3275620Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3276534Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3277852Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3279091Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3280314Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3281526Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3282742Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3284018Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3285214Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3286418Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3287636Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3288235Z dist init r=0, world=2
2022-11-23T02:44:01.3288484Z dist init r=1, world=2
2022-11-23T02:44:01.3288704Z ok (6.315s)
2022-11-23T02:44:01.3289262Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71708
2022-11-23T02:44:01.3289913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71709
2022-11-23T02:44:01.3290523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3290959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3291534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3292002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3292610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3293062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3293633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3294093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3294525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3295018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3295677Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3296368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3296873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3297343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3298222Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3298797Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3299570Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3300198Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3301116Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3302350Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3303567Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3304783Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3306001Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3307580Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3308895Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3310121Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3311332Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3312544Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3313146Z dist init r=1, world=2
2022-11-23T02:44:01.3313378Z dist init r=0, world=2
2022-11-23T02:44:01.3313617Z ok (6.415s)
2022-11-23T02:44:01.3314147Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71792
2022-11-23T02:44:01.3314832Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71793
2022-11-23T02:44:01.3315444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3315892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3316470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3316921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3317499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3317941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3318496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3318955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3319417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3319911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3320557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3321248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3321767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3322239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3323107Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3323694Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3324540Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3325113Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3326019Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3327243Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3328482Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3329705Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3330920Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3332186Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3333457Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3334668Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3335879Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3337088Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3337665Z dist init r=0, world=2
2022-11-23T02:44:01.3337910Z dist init r=1, world=2
2022-11-23T02:44:01.3338142Z ok (6.315s)
2022-11-23T02:44:01.3338670Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71876
2022-11-23T02:44:01.3339292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71877
2022-11-23T02:44:01.3339942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3340398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3340954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3341416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3341993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3342430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3342984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3343439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3343888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3344363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3345012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3345697Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3346215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3346665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3347898Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3348486Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3349281Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3349836Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3350750Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3351986Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3353207Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3354412Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3355709Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3356928Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3358146Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3359341Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3360554Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3361769Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3362430Z dist init r=1, world=2
2022-11-23T02:44:01.3362676Z dist init r=0, world=2
2022-11-23T02:44:01.3362894Z ok (6.315s)
2022-11-23T02:44:01.3363411Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71960
2022-11-23T02:44:01.3364007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71961
2022-11-23T02:44:01.3364623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3365055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3365629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3366093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3366654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3367093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3367659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3368122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3368558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3369046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3369701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3370387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3370889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3371358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3372275Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3372866Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3373649Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3374219Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3375129Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3376377Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3377597Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3378802Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3380085Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3381307Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3382513Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3383731Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3384929Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3386139Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3386737Z dist init r=0, world=2
2022-11-23T02:44:01.3387376Z dist init r=1, world=2
2022-11-23T02:44:01.3387694Z ok (6.314s)
2022-11-23T02:44:01.3388252Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72044
2022-11-23T02:44:01.3388862Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72045
2022-11-23T02:44:01.3389487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3389935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3390510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3390956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3391529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3391970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3392516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3392975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3393422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3393916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3394640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3395322Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3395837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3396313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3397177Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3397759Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3398548Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3399115Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3400013Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3401242Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3402459Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3403739Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3404962Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3406180Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3407396Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3408612Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3409801Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3411100Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3411697Z dist init r=1, world=2
2022-11-23T02:44:01.3411944Z dist init r=0, world=2
2022-11-23T02:44:01.3412163Z ok (6.314s)
2022-11-23T02:44:01.3412673Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72128
2022-11-23T02:44:01.3413269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72129
2022-11-23T02:44:01.3413880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3414310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3414883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3415350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3415926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3416351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3416916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3417378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3417817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3418312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3418965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3419698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3420208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3420681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3421563Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3422199Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3422982Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3423562Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3424479Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3425704Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3427193Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3428420Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3429625Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3430834Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3432051Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3433294Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3434504Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3435788Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3436396Z dist init r=0, world=2
2022-11-23T02:44:01.3436630Z dist init r=1, world=2
2022-11-23T02:44:01.3436868Z ok (6.315s)
2022-11-23T02:44:01.3437408Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72212
2022-11-23T02:44:01.3438038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72213
2022-11-23T02:44:01.3438635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3439083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3439656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3440104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3440677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3441113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3441678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3442196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3442647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3443146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3443805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3444475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3444994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3445461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3446315Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3446898Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3447692Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3448255Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3449171Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3450381Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3451663Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3452900Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3454111Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3455332Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3456543Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3457754Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3459021Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3460222Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3460821Z dist init r=1, world=2
2022-11-23T02:44:01.3461074Z dist init r=0, world=2
2022-11-23T02:44:01.3461305Z ok (6.415s)
2022-11-23T02:44:01.3461794Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72296
2022-11-23T02:44:01.3462395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72297
2022-11-23T02:44:01.3463006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3463455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3464008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3464471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3465045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3465472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3466046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3466497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3467202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3467700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3468366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3469048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3469567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3470029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3470902Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3471478Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3472270Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3472827Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3473737Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3475049Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3476273Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3477482Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3478707Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3479914Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3481110Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3482365Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3483582Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3484787Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3485382Z dist init r=0, world=2
2022-11-23T02:44:01.3485629Z dist init r=1, world=2
2022-11-23T02:44:01.3485845Z ok (6.415s)
2022-11-23T02:44:01.3486385Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72380
2022-11-23T02:44:01.3487010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72381
2022-11-23T02:44:01.3487602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3488049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3488624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3489146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3489729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3490180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3490748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3491189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3491637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3492130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3492779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3493457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3493973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3494437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3495318Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3495883Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3496674Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3497245Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3498206Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3499451Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3500650Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3501880Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3503086Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3504282Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3505550Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3506751Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3508165Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3509651Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3510229Z dist init r=1, world=2
2022-11-23T02:44:01.3510480Z dist init r=0, world=2
2022-11-23T02:44:01.3510715Z ok (6.415s)
2022-11-23T02:44:01.3511223Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72464
2022-11-23T02:44:01.3511802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72465
2022-11-23T02:44:01.3512407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3512858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3513408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3513875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3514545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3515004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3515559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3516022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3516475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3516952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3517611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3518292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3518813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3519266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3520140Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3520715Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3521507Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3522156Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3523061Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3524306Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3525534Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3526761Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3527979Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3529182Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3530450Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3531652Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3532911Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3534126Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3534725Z dist init r=0, world=2
2022-11-23T02:44:01.3534977Z dist init r=1, world=2
2022-11-23T02:44:01.3535197Z ok (6.315s)
2022-11-23T02:44:01.3535741Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72548
2022-11-23T02:44:01.3536445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72549
2022-11-23T02:44:01.3537058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3537487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3538067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3538531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3539089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3539528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3540095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3540552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3540992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3541482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3542138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3542833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3543328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3543795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3545065Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3545844Z   world_indices[
2022-11-23T02:44:01.3547239Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3548033Z   world_indices[
2022-11-23T02:44:01.3548280Z dist init r=0, world=2
2022-11-23T02:44:01.3548529Z dist init r=1, world=2
2022-11-23T02:44:01.3548751Z ok (6.215s)
2022-11-23T02:44:01.3549260Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72632
2022-11-23T02:44:01.3549857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72633
2022-11-23T02:44:01.3550482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3550912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3551476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3551940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3552494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3552932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3553575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3554033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3554464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3554954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3555601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3556286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3556786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3557249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3558242Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3559455Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3560682Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3561932Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3562551Z dist init r=1, world=2
2022-11-23T02:44:01.3562799Z dist init r=0, world=2
2022-11-23T02:44:01.3563034Z ok (6.215s)
2022-11-23T02:44:01.3563558Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72716
2022-11-23T02:44:01.3564187Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72717
2022-11-23T02:44:01.3564795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3565247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3565797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3566259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3566835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3567259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3567822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3568276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3568725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3569254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3569913Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3570598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3571121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3571575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3572836Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3573616Z   world_indices[
2022-11-23T02:44:01.3574776Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3575542Z   world_indices[
2022-11-23T02:44:01.3575766Z dist init r=0, world=2
2022-11-23T02:44:01.3576010Z dist init r=1, world=2
2022-11-23T02:44:01.3576242Z ok (6.215s)
2022-11-23T02:44:01.3576733Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72800
2022-11-23T02:44:01.3577335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72801
2022-11-23T02:44:01.3577939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3578387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3578985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3579459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3580037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3580462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3581026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3581486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3581933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3582409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3583065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3583750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3584272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3584724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3585707Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3587267Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3588514Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3589740Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3590348Z dist init r=0, world=2
2022-11-23T02:44:01.3590582Z dist init r=1, world=2
2022-11-23T02:44:01.3590812Z ok (6.215s)
2022-11-23T02:44:01.3591354Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72884
2022-11-23T02:44:01.3591969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72885
2022-11-23T02:44:01.3592579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3593023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3593595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3594045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3594621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3595140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3595705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3596168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3596616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3597113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3597753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3598439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3598959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3599435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3600683Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3601466Z   world_indices[
2022-11-23T02:44:01.3602612Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3603465Z   world_indices[
2022-11-23T02:44:01.3603707Z dist init r=0, world=2
2022-11-23T02:44:01.3603935Z dist init r=1, world=2
2022-11-23T02:44:01.3604168Z ok (6.415s)
2022-11-23T02:44:01.3604674Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72968
2022-11-23T02:44:01.3605255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72969
2022-11-23T02:44:01.3605867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3606317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3606886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3607336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3607909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3608343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3608887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3609348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3609794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3610295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3610931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3611668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3612196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3612663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3613912Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3614776Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:44:01.3616008Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3616872Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:44:01.3617195Z dist init r=0, world=2
2022-11-23T02:44:01.3617425Z dist init r=1, world=2
2022-11-23T02:44:01.3617659Z ok (6.315s)
2022-11-23T02:44:01.3618197Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73052
2022-11-23T02:44:01.3618866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73053
2022-11-23T02:44:01.3619484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3619933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3620504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3620951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3621526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3621968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3622520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3622980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3623426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3623919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3624555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3625237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3625753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3626216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3627726Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3628530Z   world_indices[
2022-11-23T02:44:01.3629682Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3630465Z   world_indices[
2022-11-23T02:44:01.3630709Z dist init r=0, world=2
2022-11-23T02:44:01.3630938Z dist init r=1, world=2
2022-11-23T02:44:01.3631170Z ok (6.315s)
2022-11-23T02:44:01.3631678Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73136
2022-11-23T02:44:01.3632259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73137
2022-11-23T02:44:01.3632899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3633345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3633922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3634374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3635032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3635472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3636026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3636488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3636937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3637426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3638064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3638745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3639267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3639737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3640984Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3641856Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:44:01.3643081Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3643984Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:44:01.3644320Z dist init r=1, world=2
2022-11-23T02:44:01.3644553Z dist init r=0, world=2
2022-11-23T02:44:01.3644784Z ok (6.315s)
2022-11-23T02:44:01.3645341Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73220
2022-11-23T02:44:01.3645985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73221
2022-11-23T02:44:01.3646582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3647032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3647599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3648055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3648623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3649061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3649625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3650068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3650517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3651089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3651734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3652424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3652941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3653406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3654261Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3654842Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3655640Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3656209Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3657130Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3658356Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3659586Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3660852Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3662083Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3663313Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3664536Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3665757Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3667196Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3668417Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3669014Z dist init r=0, world=2
2022-11-23T02:44:01.3669261Z dist init r=1, world=2
2022-11-23T02:44:01.3669494Z ok (6.315s)
2022-11-23T02:44:01.3670003Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73304
2022-11-23T02:44:01.3670624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73305
2022-11-23T02:44:01.3671239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3671692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3672252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3672717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3673293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3673717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3674284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3674745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3675198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3675671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3676403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3677106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3677625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3678070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3678938Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3679523Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3680325Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3680884Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3681799Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3683025Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3684330Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3685540Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3686758Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3687970Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3689172Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3690383Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3691655Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3692876Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3693469Z dist init r=1, world=2
2022-11-23T02:44:01.3693719Z dist init r=0, world=2
2022-11-23T02:44:01.3693940Z ok (6.315s)
2022-11-23T02:44:01.3694498Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73388
2022-11-23T02:44:01.3695141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73389
2022-11-23T02:44:01.3695740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3696190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3696761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3697219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3697777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3698278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3698849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3699294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3699748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3700245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3700896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3701570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3702083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3702554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3703430Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3703990Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3704783Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3705354Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3706267Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3707823Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3709048Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3710274Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3711495Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3712699Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3713904Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3715192Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3716420Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3717634Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3718217Z dist init r=1, world=2
2022-11-23T02:44:01.3718467Z dist init r=0, world=2
2022-11-23T02:44:01.3718704Z ok (6.214s)
2022-11-23T02:44:01.3719236Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73472
2022-11-23T02:44:01.3719838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73473
2022-11-23T02:44:01.3720445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3720891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3721448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3721918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3722549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3722988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3723596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3724066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3724514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3725005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3725647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3726338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3726854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3727306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3728187Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3728764Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3729554Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3730175Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3731071Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3732307Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3733560Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3734784Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3735996Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3737206Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3738415Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3739659Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3740875Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3742090Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3742696Z dist init r=1, world=2
2022-11-23T02:44:01.3742947Z dist init r=0, world=2
2022-11-23T02:44:01.3743166Z ok (6.315s)
2022-11-23T02:44:01.3743721Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73556
2022-11-23T02:44:01.3744361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73557
2022-11-23T02:44:01.3744967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3745452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3746026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3746492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3747248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3747696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3748264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3748723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3749156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3749650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3750312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3751000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3751504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3751977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3752851Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3753430Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3754207Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3754781Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3755767Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3757017Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3758242Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3759475Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3760672Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3761886Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3763197Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3764404Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3765613Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3766824Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3767423Z dist init r=1, world=2
2022-11-23T02:44:01.3767655Z dist init r=0, world=2
2022-11-23T02:44:01.3767889Z ok (6.414s)
2022-11-23T02:44:01.3768415Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73640
2022-11-23T02:44:01.3769013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73641
2022-11-23T02:44:01.3769621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3770066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3770690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3771145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3771725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3772167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3772736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3773177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3773632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3774120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3774763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3775448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3775964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3776431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3777291Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3777934Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3778737Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3779312Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3780209Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3781441Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3782676Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3783901Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3785116Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3786380Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3787846Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3789063Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3790289Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3791478Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3792083Z dist init r=1, world=2
2022-11-23T02:44:01.3792336Z dist init r=0, world=2
2022-11-23T02:44:01.3792574Z ok (6.314s)
2022-11-23T02:44:01.3793205Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73724
2022-11-23T02:44:01.3793849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73725
2022-11-23T02:44:01.3794466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3794914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3795463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3795921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3796492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3796910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3797483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3797938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3798393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3798871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3799524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3800208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3800727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3801182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3802056Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3802695Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3803502Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3804054Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3804964Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3806202Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3807423Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3808636Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3809926Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3811129Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3812335Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3813557Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3814763Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3815965Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3816564Z dist init r=1, world=2
2022-11-23T02:44:01.3816794Z dist init r=0, world=2
2022-11-23T02:44:01.3817029Z ok (6.415s)
2022-11-23T02:44:01.3817608Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73808
2022-11-23T02:44:01.3818233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73809
2022-11-23T02:44:01.3818828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3819273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3819842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3820309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3820876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3821315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3821888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3822332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3822781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3823272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3823925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3824603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3825177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3825645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3826529Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3827267Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3828080Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3828651Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3829571Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3830809Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3832013Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3833269Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3834569Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3835793Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3837010Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3838229Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3839440Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3840729Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3841308Z dist init r=1, world=2
2022-11-23T02:44:01.3841557Z dist init r=0, world=2
2022-11-23T02:44:01.3841792Z ok (6.315s)
2022-11-23T02:44:01.3842324Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73892
2022-11-23T02:44:01.3842958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73893
2022-11-23T02:44:01.3843565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3844013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3844575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3845041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3845622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3846062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3846610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3847068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3847518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3847994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3848649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3849330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3849898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3850358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3851616Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3852494Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:44:01.3853730Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3854582Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:44:01.3854891Z dist init r=0, world=2
2022-11-23T02:44:01.3855137Z dist init r=1, world=2
2022-11-23T02:44:01.3855371Z ok (6.315s)
2022-11-23T02:44:01.3855889Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73976
2022-11-23T02:44:01.3856539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73977
2022-11-23T02:44:01.3856918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3857092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3857472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3857665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3858028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3858201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3858558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3858750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3858997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3859231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3859635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3860027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3860252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3860479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3861223Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3862021Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3862773Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3864975Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3865101Z dist init r=1, world=2
2022-11-23T02:44:01.3865195Z dist init r=0, world=2
2022-11-23T02:44:01.3865295Z ok (6.215s)
2022-11-23T02:44:01.3865727Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74060
2022-11-23T02:44:01.3865945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74061
2022-11-23T02:44:01.3866336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3866510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3867107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3867429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3867794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3867975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3868349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3868540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3868783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3869019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3869416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3869814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3870041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3870253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3871270Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3871472Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:44:01.3872552Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3872757Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:44:01.3872869Z dist init r=1, world=2
2022-11-23T02:44:01.3872977Z dist init r=0, world=2
2022-11-23T02:44:01.3873110Z ok (6.315s)
2022-11-23T02:44:01.3873509Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74144
2022-11-23T02:44:01.3873725Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74145
2022-11-23T02:44:01.3874103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3874260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3874641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3874830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3875193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3875362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3875736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3875925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3876227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3876446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3876848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3877242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3877469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3877695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3878439Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3879182Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3879911Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3880632Z [W python_variable.cpp:318] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2022-11-23T02:44:01.3880746Z dist init r=0, world=2
2022-11-23T02:44:01.3880854Z dist init r=1, world=2
2022-11-23T02:44:01.3880952Z ok (6.214s)
2022-11-23T02:44:01.3881406Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74228
2022-11-23T02:44:01.3881632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74229
2022-11-23T02:44:01.3882006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3882178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3882555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3882748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3883114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3883287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3883661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3883831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3884074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3884307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3884703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3885152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3885382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3885611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3886639Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3886748Z   world_indices[
2022-11-23T02:44:01.3887769Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3887878Z   world_indices[
2022-11-23T02:44:01.3887974Z dist init r=0, world=2
2022-11-23T02:44:01.3888081Z dist init r=1, world=2
2022-11-23T02:44:01.3888177Z ok (6.415s)
2022-11-23T02:44:01.3888565Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74312
2022-11-23T02:44:01.3888780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74313
2022-11-23T02:44:01.3889149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3889326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3889702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3889874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3890285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3890466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3890844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3891033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3891277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3891514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3891915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3892307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3892520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3892744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3893767Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3894039Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:44:01.3895062Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3895261Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:44:01.3895371Z dist init r=0, world=2
2022-11-23T02:44:01.3895480Z dist init r=1, world=2
2022-11-23T02:44:01.3895580Z ok (6.315s)
2022-11-23T02:44:01.3896000Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74396
2022-11-23T02:44:01.3896219Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74397
2022-11-23T02:44:01.3896573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3896749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3897124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3897310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3897673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3897845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3898217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3898411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3898653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3898922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3899330Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3899722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3899949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3900174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3901200Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3901308Z   world_indices[
2022-11-23T02:44:01.3902326Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_exec_order_utils.py:261: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3902429Z   world_indices[
2022-11-23T02:44:01.3902589Z dist init r=0, world=2
2022-11-23T02:44:01.3902698Z dist init r=1, world=2
2022-11-23T02:44:01.3902780Z ok (6.314s)
2022-11-23T02:44:01.3903172Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74480
2022-11-23T02:44:01.3903394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74481
2022-11-23T02:44:01.3903762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3903935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3904309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3904498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3904861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3905021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3905393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3905583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3905826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3906059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3906455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3907015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3907259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3907487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3908592Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3908787Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:44:01.3909808Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/flat_param.py:1255: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:318.)
2022-11-23T02:44:01.3910012Z   _ext_post_unflatten_transform(subtensor.view(shape), param_extension)
2022-11-23T02:44:01.3910127Z dist init r=1, world=2
2022-11-23T02:44:01.3910236Z dist init r=0, world=2
2022-11-23T02:44:01.3910334Z ok (6.214s)
2022-11-23T02:44:01.3910694Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74564
2022-11-23T02:44:01.3910911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74565
2022-11-23T02:44:01.3911280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3911454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3911892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3912083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3912451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3912624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3912997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3913187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3913428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3913661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3914045Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3914438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3914668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3914893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3915522Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3915665Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3936513Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3936729Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3936848Z dist init r=0, world=2
2022-11-23T02:44:01.3936959Z dist init r=1, world=2
2022-11-23T02:44:01.3937041Z ok (5.914s)
2022-11-23T02:44:01.3937381Z   test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded)
2022-11-23T02:44:01.3937624Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.001s)
2022-11-23T02:44:01.3937982Z   test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74648
2022-11-23T02:44:01.3938204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74649
2022-11-23T02:44:01.3938622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3938806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3939192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3939386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3939739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3939917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3940292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3940481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3940730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3940973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3941467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3941865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3942081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3942313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3942428Z dist init r=0, world=2
2022-11-23T02:44:01.3942539Z dist init r=1, world=2
2022-11-23T02:44:01.3942639Z ok (6.216s)
2022-11-23T02:44:01.3942991Z   test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74732
2022-11-23T02:44:01.3943210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74733
2022-11-23T02:44:01.3943588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3943746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3944131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3944325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3944689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3944860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3945237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3945426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3945678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3945923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3946360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3946976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3947227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3947457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3948093Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:44:01.3948213Z   warnings.warn(
2022-11-23T02:44:01.3948832Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2455: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-11-23T02:44:01.3948947Z   warnings.warn(
2022-11-23T02:44:01.3949061Z dist init r=1, world=2
2022-11-23T02:44:01.3949150Z dist init r=0, world=2
2022-11-23T02:44:01.3949250Z ok (5.916s)
2022-11-23T02:44:01.3949589Z   test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74816
2022-11-23T02:44:01.3949810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74817
2022-11-23T02:44:01.3950183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3950460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3950846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3951038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3951384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3951561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3951935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3952125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3952371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3952614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3953020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3953417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3953650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3953860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3953972Z dist init r=1, world=2
2022-11-23T02:44:01.3954082Z dist init r=0, world=2
2022-11-23T02:44:01.3954182Z ok (6.014s)
2022-11-23T02:44:01.3954537Z   test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74900
2022-11-23T02:44:01.3954756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74901
2022-11-23T02:44:01.3955134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3955309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3955736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3955937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3956303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3956476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3956848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3957036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3957286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3957534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3957935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3958311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3958540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3958769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3958880Z dist init r=1, world=2
2022-11-23T02:44:01.3958991Z dist init r=0, world=2
2022-11-23T02:44:01.3959094Z ok (6.014s)
2022-11-23T02:44:01.3959453Z   test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74984
2022-11-23T02:44:01.3959732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74985
2022-11-23T02:44:01.3960091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3960272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3960655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3960847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3961208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3961382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3961754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3961946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3962174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3962423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3962824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3963216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3963445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3963674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3963787Z dist init r=0, world=2
2022-11-23T02:44:01.3963903Z dist init r=1, world=2
2022-11-23T02:44:01.3963984Z ok (6.014s)
2022-11-23T02:44:01.3964324Z   test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75068
2022-11-23T02:44:01.3964542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75069
2022-11-23T02:44:01.3964965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3965147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3965528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3965723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3966085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3966263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3966619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3966809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3967057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3967303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-11-23T02:44:01.3967701Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3968094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-11-23T02:44:01.3968323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3968610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T02:44:01.3968724Z dist init r=0, world=2
2022-11-23T02:44:01.3968815Z dist init r=1, world=2
2022-11-23T02:44:01.3968915Z ok (6.315s)
2022-11-23T02:44:01.3969268Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75152
2022-11-23T02:44:01.3969641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3969817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3970194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3970386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3970631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3971015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:44:01.3971243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3971354Z dist init r=0, world=1
2022-11-23T02:44:01.3971455Z ok (4.510s)
2022-11-23T02:44:01.3971813Z   test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75194
2022-11-23T02:44:01.3972181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3972355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3972730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3972902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3973152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3973549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:44:01.3973839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3974488Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3974632Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3974744Z dist init r=0, world=1
2022-11-23T02:44:01.3974844Z ok (4.610s)
2022-11-23T02:44:01.3975212Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75236
2022-11-23T02:44:01.3975568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T02:44:01.3975742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T02:44:01.3976122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T02:44:01.3976312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T02:44:01.3976557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-11-23T02:44:01.3976953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2022-11-23T02:44:01.3977183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T02:44:01.3977812Z /opt/conda/lib/python3.10/site-packages/torch/storage.py:315: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.
2022-11-23T02:44:01.3978007Z   warnings.warn(message, UserWarning)
2022-11-23T02:44:01.3978100Z dist init r=0, world=1
2022-11-23T02:44:01.3978200Z ok (4.610s)
2022-11-23T02:44:01.3978226Z 
2022-11-23T02:44:01.3978498Z ----------------------------------------------------------------------
2022-11-23T02:44:01.3978616Z Ran 52 tests in 314.754s
2022-11-23T02:44:01.3978635Z 
2022-11-23T02:44:01.3978743Z OK (skipped=1)
2022-11-23T02:44:01.3978761Z 
2022-11-23T02:44:01.3978885Z Generating XML reports...
2022-11-23T02:44:01.3979397Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221123023846.xml
2022-11-23T02:44:01.3979903Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221123023846.xml
2022-11-23T02:44:01.3979926Z 
2022-11-23T02:44:01.3980366Z ##[endgroup]
2022-11-23T02:44:01.3980875Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_pazbq4_p)
2022-11-23T02:44:01.3980895Z 
2022-11-23T02:44:01.3981193Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2022-11-23 02:44:01.312170]
2022-11-23T02:44:01.3981721Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-11-23 02:44:01.312443]
2022-11-23T03:04:41.7988999Z 
2022-11-23T03:04:41.7991979Z Expand the folded group to see the log file of distributed/rpc/cuda/test_tensorpipe_agent
2022-11-23T03:04:41.7993612Z ##[group]PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_gdovk1ur)
2022-11-23T03:04:41.7994770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9su5m90a
2022-11-23T03:04:41.7995373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9su5m90a/_remote_module_non_scriptable.py
2022-11-23T03:04:41.7996458Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote_gpu>]>
2022-11-23T03:04:41.7997479Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest)
2022-11-23T03:04:41.7998997Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_simple>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]>
2022-11-23T03:04:41.7999759Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest)
2022-11-23T03:04:41.8000181Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest)
2022-11-23T03:04:41.8000629Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest)
2022-11-23T03:04:41.8001557Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]>
2022-11-23T03:04:41.8002960Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-11-23T03:04:41.8003836Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest)
2022-11-23T03:04:41.8004766Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest)
2022-11-23T03:04:41.8005576Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest)
2022-11-23T03:04:41.8006139Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRpcTest testMethod=test_profiler_remote_cuda>]>
2022-11-23T03:04:41.8006830Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest)
2022-11-23T03:04:41.8008123Z <unittest.suite.TestSuite tests=[<__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]>
2022-11-23T03:04:41.8010081Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-11-23T03:04:41.8010841Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-11-23T03:04:41.8011669Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-11-23T03:04:41.8012314Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-11-23T03:04:41.8012739Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2022-11-23T03:04:41.8013407Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2022-11-23T03:04:41.8014063Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2022-11-23T03:04:41.8014768Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2022-11-23T03:04:41.8032068Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_nested_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]>
2022-11-23T03:04:41.8048943Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8049593Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8050195Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8050870Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8051369Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8052168Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8052930Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8053645Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8054352Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8055012Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8055522Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8056172Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8056801Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8057457Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8058025Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8058636Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8059275Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8059758Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8060358Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8060993Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8061653Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8062189Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8062833Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8063505Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8064163Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8064710Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8065307Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8065923Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8066540Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8068563Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8069118Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8069678Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8070293Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8070917Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8071433Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8072103Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8072698Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8073308Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8073795Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8074460Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8075070Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8075585Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8076308Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8077003Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8077657Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8078167Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8078787Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8079421Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8079956Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8080671Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8081284Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8081858Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8082454Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8083106Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8083770Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8084313Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8085001Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8085856Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8086498Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8087078Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8087644Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8088244Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8088844Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8089363Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8090022Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8090620Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8091124Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8091782Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8092445Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8093106Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8093644Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8094313Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8094949Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8095512Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8096103Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8096722Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8097288Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8097892Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8098529Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8099212Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8099768Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8100420Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8101054Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8101608Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8102185Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8102830Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8103449Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2022-11-23T03:04:41.8104534Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_device_maps_backward_pass>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]>
2022-11-23T03:04:41.8105626Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-11-23T03:04:41.8106312Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-11-23T03:04:41.8107611Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2022-11-23T03:04:41.8108369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8108940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8109529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8109999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8110447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ks1mm3y
2022-11-23T03:04:41.8111029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ks1mm3y/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8111345Z 
2022-11-23T03:04:41.8111448Z Running tests...
2022-11-23T03:04:41.8111866Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8112426Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8113110Z   test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8113762Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75346
2022-11-23T03:04:41.8114475Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75347
2022-11-23T03:04:41.8115074Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75348
2022-11-23T03:04:41.8115792Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75349
2022-11-23T03:04:41.8116701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8117411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8118296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8118895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8119614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8120320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8121163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8121680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8122404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8123018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8123752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8124232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8124963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8125549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8126155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8126820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8127485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzm_y5piu
2022-11-23T03:04:41.8128043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzm_y5piu/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8128576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsysbv_p9
2022-11-23T03:04:41.8129092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsysbv_p9/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8129615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqv0ew5n6
2022-11-23T03:04:41.8130242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqv0ew5n6/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8130765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqnusmv3c
2022-11-23T03:04:41.8131283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqnusmv3c/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8131787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8132254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8132702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8133171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8133555Z skip: Need at least 4 CUDA devices (4.318s)
2022-11-23T03:04:41.8133752Z 
2022-11-23T03:04:41.8134048Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8134367Z Ran 1 test in 4.318s
2022-11-23T03:04:41.8134520Z 
2022-11-23T03:04:41.8134636Z OK (skipped=1)
2022-11-23T03:04:41.8134785Z 
2022-11-23T03:04:41.8134910Z Generating XML reports...
2022-11-23T03:04:41.8135571Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20221123024405.xml
2022-11-23T03:04:41.8136357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8136812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8137386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8137845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8138311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ei2et5j
2022-11-23T03:04:41.8138857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ei2et5j/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8139159Z 
2022-11-23T03:04:41.8139258Z Running tests...
2022-11-23T03:04:41.8139646Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8140295Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8140851Z   test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8141306Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75517
2022-11-23T03:04:41.8141756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75518
2022-11-23T03:04:41.8142191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75519
2022-11-23T03:04:41.8142636Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75520
2022-11-23T03:04:41.8143231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8143681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8144259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8144730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8145293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8145738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8146307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8147298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8148067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8148510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8149079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8149529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8150104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8150544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8151093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8151557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8152022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0q_ccu2
2022-11-23T03:04:41.8152566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0q_ccu2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8153081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphfm2rlv_
2022-11-23T03:04:41.8153620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphfm2rlv_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8154149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmeqios3b
2022-11-23T03:04:41.8154679Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmeqios3b/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8155188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbungqzd6
2022-11-23T03:04:41.8155721Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbungqzd6/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8156225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8156683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8157149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8157692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8158096Z fi_getinfo: -61
2022-11-23T03:04:41.8158352Z fi_getinfo: -61
2022-11-23T03:04:41.8158622Z fi_getinfo: -61
2022-11-23T03:04:41.8158890Z fi_getinfo: -61
2022-11-23T03:04:41.8159103Z ok (6.636s)
2022-11-23T03:04:41.8159251Z 
2022-11-23T03:04:41.8159516Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8159842Z Ran 1 test in 6.637s
2022-11-23T03:04:41.8160002Z 
2022-11-23T03:04:41.8160077Z OK
2022-11-23T03:04:41.8160207Z 
2022-11-23T03:04:41.8160331Z Generating XML reports...
2022-11-23T03:04:41.8161011Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123024412.xml
2022-11-23T03:04:41.8161781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8162215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8162795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8163264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8163713Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtc8virr
2022-11-23T03:04:41.8164251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtc8virr/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8164550Z 
2022-11-23T03:04:41.8164659Z Running tests...
2022-11-23T03:04:41.8165065Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8165688Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8166255Z   test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8166768Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75940
2022-11-23T03:04:41.8167199Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75941
2022-11-23T03:04:41.8167645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75942
2022-11-23T03:04:41.8168082Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75943
2022-11-23T03:04:41.8168688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8169117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8169692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8170161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8170722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8171167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8171737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8172197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8172752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8173197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8173769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8174234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8174794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8175236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8175858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8176310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8176772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfyu2il_q
2022-11-23T03:04:41.8177316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfyu2il_q/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8177850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgq9avps
2022-11-23T03:04:41.8178373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgq9avps/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8178905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6df7b2k
2022-11-23T03:04:41.8179437Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6df7b2k/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8179947Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrm2qow2
2022-11-23T03:04:41.8180479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrm2qow2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8180985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8181454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8181905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8182364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8182816Z fi_getinfo: -61
2022-11-23T03:04:41.8183071Z fi_getinfo: -61
2022-11-23T03:04:41.8183345Z fi_getinfo: -61
2022-11-23T03:04:41.8183613Z fi_getinfo: -61
2022-11-23T03:04:41.8183830Z ok (6.746s)
2022-11-23T03:04:41.8183975Z 
2022-11-23T03:04:41.8184246Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8184575Z Ran 1 test in 6.746s
2022-11-23T03:04:41.8184734Z 
2022-11-23T03:04:41.8184827Z OK
2022-11-23T03:04:41.8184941Z 
2022-11-23T03:04:41.8185065Z Generating XML reports...
2022-11-23T03:04:41.8185743Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123024422.xml
2022-11-23T03:04:41.8186504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8187656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8188264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8188738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8189201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qr1585l
2022-11-23T03:04:41.8189720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qr1585l/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8190018Z 
2022-11-23T03:04:41.8190126Z Running tests...
2022-11-23T03:04:41.8190533Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8191103Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8191660Z   test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8192180Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76363
2022-11-23T03:04:41.8192632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76364
2022-11-23T03:04:41.8193055Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76365
2022-11-23T03:04:41.8193486Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76366
2022-11-23T03:04:41.8194185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8194653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8195212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8195676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8196258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8196688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8197255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8197715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8198292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8198714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8199281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8199743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8200317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8200737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8201384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8201845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8202296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44x76rr8
2022-11-23T03:04:41.8202832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44x76rr8/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8203362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7pqsrfnu
2022-11-23T03:04:41.8203894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7pqsrfnu/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8204403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hynxist
2022-11-23T03:04:41.8204932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hynxist/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8205465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntdkytj5
2022-11-23T03:04:41.8205974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntdkytj5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8206483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8206961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8207426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8207873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8208257Z fi_getinfo: -61
2022-11-23T03:04:41.8208531Z fi_getinfo: -61
2022-11-23T03:04:41.8208784Z fi_getinfo: -61
2022-11-23T03:04:41.8209053Z fi_getinfo: -61
2022-11-23T03:04:41.8209288Z ok (6.731s)
2022-11-23T03:04:41.8209433Z 
2022-11-23T03:04:41.8209689Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8210019Z Ran 1 test in 6.731s
2022-11-23T03:04:41.8210180Z 
2022-11-23T03:04:41.8210274Z OK
2022-11-23T03:04:41.8210407Z 
2022-11-23T03:04:41.8210568Z Generating XML reports...
2022-11-23T03:04:41.8211288Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123024431.xml
2022-11-23T03:04:41.8212063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8212512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8213064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8213537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8214003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxc8jz5f
2022-11-23T03:04:41.8214548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxc8jz5f/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8214847Z 
2022-11-23T03:04:41.8214937Z Running tests...
2022-11-23T03:04:41.8215338Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8215920Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8216473Z   test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8216986Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76786
2022-11-23T03:04:41.8217434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76787
2022-11-23T03:04:41.8218040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8218531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8219105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8219573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8220153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8220575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8221141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8221605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8222050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpruh0k9br
2022-11-23T03:04:41.8222586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpruh0k9br/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8223118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprt1qno2p
2022-11-23T03:04:41.8223647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprt1qno2p/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8224141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8224612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8225004Z fi_getinfo: -61
2022-11-23T03:04:41.8225266Z fi_getinfo: -61
2022-11-23T03:04:41.8225498Z ok (6.050s)
2022-11-23T03:04:41.8225646Z 
2022-11-23T03:04:41.8225910Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8226225Z Ran 1 test in 6.050s
2022-11-23T03:04:41.8226402Z 
2022-11-23T03:04:41.8226471Z OK
2022-11-23T03:04:41.8226602Z 
2022-11-23T03:04:41.8226728Z Generating XML reports...
2022-11-23T03:04:41.8227837Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024441.xml
2022-11-23T03:04:41.8228594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8229043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8229702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8230181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8230630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5jsf2_k
2022-11-23T03:04:41.8231174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5jsf2_k/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8231476Z 
2022-11-23T03:04:41.8231583Z Running tests...
2022-11-23T03:04:41.8231973Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8232545Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8233124Z   test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8233652Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76976
2022-11-23T03:04:41.8234089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76977
2022-11-23T03:04:41.8234692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8235143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8235695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8236166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8236870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8237315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8237866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8238337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8238804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfklxr7dt
2022-11-23T03:04:41.8239345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfklxr7dt/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8239857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp83dvuziu
2022-11-23T03:04:41.8240388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83dvuziu/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8240897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8241347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8241733Z fi_getinfo: -61
2022-11-23T03:04:41.8242005Z fi_getinfo: -61
2022-11-23T03:04:41.8242494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfklxr7dt/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T03:04:41.8243228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp83dvuziu/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T03:04:41.8243968Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmp83dvuziu/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2022-11-23T03:04:41.8244443Z ok (6.228s)
2022-11-23T03:04:41.8244594Z 
2022-11-23T03:04:41.8244868Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8245175Z Ran 1 test in 6.228s
2022-11-23T03:04:41.8245335Z 
2022-11-23T03:04:41.8245427Z OK
2022-11-23T03:04:41.8245560Z 
2022-11-23T03:04:41.8245682Z Generating XML reports...
2022-11-23T03:04:41.8246390Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024449.xml
2022-11-23T03:04:41.8247165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8247618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8248194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8248646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8249112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp41k5ueqo
2022-11-23T03:04:41.8249659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp41k5ueqo/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8249958Z 
2022-11-23T03:04:41.8250050Z Running tests...
2022-11-23T03:04:41.8250454Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8251033Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8251588Z   test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8252070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77182
2022-11-23T03:04:41.8252521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77183
2022-11-23T03:04:41.8253128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8253558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8254221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8254690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8255271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8255695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8256261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8256721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8257188Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe24czcdd
2022-11-23T03:04:41.8257710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe24czcdd/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8258244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn11oxx0_
2022-11-23T03:04:41.8258777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn11oxx0_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8259263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8259737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8260131Z fi_getinfo: -61
2022-11-23T03:04:41.8260408Z fi_getinfo: -61
2022-11-23T03:04:41.8260657Z On WorkerInfo(id=1, name=worker1):
2022-11-23T03:04:41.8278707Z RuntimeError('CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f662136159b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f662135cdfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7f66215e706e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #3: <unknown function> + 0x17c9d (0x7f66215bfc9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #4: <unknown function> + 0xdf3ecd (0x7f66225ffecd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #5: <unknown function> + 0x29c7ab5 (0x7f66241d3ab5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #6: <unknown function> + 0x29c7c5b (0x7f66241d3c5b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f662e3d6cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: <unknown function> + 0x20c43b5 (0x7f662e6f83b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f662e411e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x12691af (0x7f662d89d1af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7f662dc33641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x22a6e23 (0x7f662e8dae23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f662e0f70e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x20c8908 (0x7f662e6fc908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f662e0f70e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: <unknown function> + 0x3443e11 (0x7f662fa77e11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x34443bb (0x7f662fa783bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f662e150de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f662dc2b05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #20: <unknown function> + 0x2471e09 (0x7f662eaa5e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f662e2c23f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #22: <unknown function> + 0x36467f (0x7f663937767f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #23: <unknown function> + 0x364b3c (0x7f6639377b3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #24: <unknown function> + 0x1ddc68 (0x55b088505c68 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0x1049f3 (0x55b08842c9f3 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0x104425 (0x55b08842c425 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x18fc9b (0x55b0884b7c9b in /opt/conda/bin/python)\nframe #30: <unknown function> + 0x1052a5 (0x55b08842d2a5 in /opt/conda/bin/python)\nframe #31: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #32: <unknown function> + 0x18fc9b (0x55b0884b7c9b in /opt/conda/bin/python)\nframe #33: <unknown function> + 0x1052a5 (0x55b08842d2a5 in /opt/conda/bin/python)\nframe #34: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x20a (0x55b08846ffaa in /opt/conda/bin/python)\nframe #36: _PyEval_EvalFrameDefault + 0x26e4 (0x55b08850b774 in /opt/conda/bin/python)\nframe #37: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #38: _PyObject_Call + 0x20a (0x55b08846ffaa in /opt/conda/bin/python)\nframe #39: <unknown function> + 0xaa8dba (0x7f6639abbdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f6639ab9ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f6639abd2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f6639abd9a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f663105f5c4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f6639abd0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #45: <unknown function> + 0x4a249c3 (0x7f66310589c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f6631059558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f6631053857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #48: <unknown function> + 0x4a54542 (0x7f6631088542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f662134f90b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #50: <unknown function> + 0xdbbf4 (0x7f66511b5bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #51: <unknown function> + 0x76db (0x7f66718096db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #52: clone + 0x3f (0x7f667153261f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T03:04:41.8289239Z Traceback (most recent call last):
2022-11-23T03:04:41.8289770Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T03:04:41.8290226Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T03:04:41.8290806Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 92, in _create_module
2022-11-23T03:04:41.8291196Z     module.to(device)
2022-11-23T03:04:41.8291640Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1120, in to
2022-11-23T03:04:41.8292004Z     return self._apply(convert)
2022-11-23T03:04:41.8292478Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 795, in _apply
2022-11-23T03:04:41.8292835Z     param_applied = fn(param)
2022-11-23T03:04:41.8293317Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1118, in convert
2022-11-23T03:04:41.8293781Z     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2022-11-23T03:04:41.8294174Z RuntimeError: CUDA error: invalid device ordinal
2022-11-23T03:04:41.8294604Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-11-23T03:04:41.8295054Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-11-23T03:04:41.8295620Z Exception raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):
2022-11-23T03:04:41.8296483Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f662136159b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.8297442Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f662135cdfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.8298328Z frame #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7f66215e706e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T03:04:41.8299019Z frame #3: <unknown function> + 0x17c9d (0x7f66215bfc9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T03:04:41.8299664Z frame #4: <unknown function> + 0xdf3ecd (0x7f66225ffecd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.8300295Z frame #5: <unknown function> + 0x29c7ab5 (0x7f66241d3ab5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.8300933Z frame #6: <unknown function> + 0x29c7c5b (0x7f66241d3c5b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.8301998Z frame #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f662e3d6cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8302842Z frame #8: <unknown function> + 0x20c43b5 (0x7f662e6f83b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8303799Z frame #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f662e411e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8304641Z frame #10: <unknown function> + 0x12691af (0x7f662d89d1af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8305609Z frame #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7f662dc33641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8306412Z frame #12: <unknown function> + 0x22a6e23 (0x7f662e8dae23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8307853Z frame #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f662e0f70e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8308706Z frame #14: <unknown function> + 0x20c8908 (0x7f662e6fc908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8309695Z frame #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f662e0f70e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8310532Z frame #16: <unknown function> + 0x3443e11 (0x7f662fa77e11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8311200Z frame #17: <unknown function> + 0x34443bb (0x7f662fa783bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8312252Z frame #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f662e150de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8313362Z frame #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f662dc2b05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8314167Z frame #20: <unknown function> + 0x2471e09 (0x7f662eaa5e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8315126Z frame #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f662e2c23f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8315958Z frame #22: <unknown function> + 0x36467f (0x7f663937767f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8316608Z frame #23: <unknown function> + 0x364b3c (0x7f6639377b3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8317067Z frame #24: <unknown function> + 0x1ddc68 (0x55b088505c68 in /opt/conda/bin/python)
2022-11-23T03:04:41.8317450Z frame #25: <unknown function> + 0x1049f3 (0x55b08842c9f3 in /opt/conda/bin/python)
2022-11-23T03:04:41.8317844Z frame #26: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8318240Z frame #27: <unknown function> + 0x104425 (0x55b08842c425 in /opt/conda/bin/python)
2022-11-23T03:04:41.8318620Z frame #28: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8318992Z frame #29: <unknown function> + 0x18fc9b (0x55b0884b7c9b in /opt/conda/bin/python)
2022-11-23T03:04:41.8319387Z frame #30: <unknown function> + 0x1052a5 (0x55b08842d2a5 in /opt/conda/bin/python)
2022-11-23T03:04:41.8319783Z frame #31: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8320219Z frame #32: <unknown function> + 0x18fc9b (0x55b0884b7c9b in /opt/conda/bin/python)
2022-11-23T03:04:41.8320619Z frame #33: <unknown function> + 0x1052a5 (0x55b08842d2a5 in /opt/conda/bin/python)
2022-11-23T03:04:41.8321003Z frame #34: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8321388Z frame #35: _PyObject_Call + 0x20a (0x55b08846ffaa in /opt/conda/bin/python)
2022-11-23T03:04:41.8321781Z frame #36: _PyEval_EvalFrameDefault + 0x26e4 (0x55b08850b774 in /opt/conda/bin/python)
2022-11-23T03:04:41.8322186Z frame #37: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8322569Z frame #38: _PyObject_Call + 0x20a (0x55b08846ffaa in /opt/conda/bin/python)
2022-11-23T03:04:41.8323152Z frame #39: <unknown function> + 0xaa8dba (0x7f6639abbdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8323945Z frame #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f6639ab9ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8324950Z frame #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f6639abd2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8326076Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f6639abd9a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8327366Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f663105f5c4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8328630Z frame #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f6639abd0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8329491Z frame #45: <unknown function> + 0x4a249c3 (0x7f66310589c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8330418Z frame #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f6631059558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8331484Z frame #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f6631053857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8332250Z frame #48: <unknown function> + 0x4a54542 (0x7f6631088542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8332918Z frame #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f662134f90b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.8333401Z frame #50: <unknown function> + 0xdbbf4 (0x7f66511b5bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T03:04:41.8333947Z frame #51: <unknown function> + 0x76db (0x7f66718096db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T03:04:41.8334452Z frame #52: clone + 0x3f (0x7f667153261f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T03:04:41.8334674Z 
2022-11-23T03:04:41.8334692Z 
2022-11-23T03:04:41.8334829Z On WorkerInfo(id=1, name=worker1):
2022-11-23T03:04:41.8372307Z RuntimeError('On WorkerInfo(id=1, name=worker1):\nRuntimeError(\'CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f662136159b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f662135cdfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7f66215e706e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #3: <unknown function> + 0x17c9d (0x7f66215bfc9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #4: <unknown function> + 0xdf3ecd (0x7f66225ffecd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #5: <unknown function> + 0x29c7ab5 (0x7f66241d3ab5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #6: <unknown function> + 0x29c7c5b (0x7f66241d3c5b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f662e3d6cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: <unknown function> + 0x20c43b5 (0x7f662e6f83b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f662e411e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x12691af (0x7f662d89d1af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7f662dc33641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x22a6e23 (0x7f662e8dae23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f662e0f70e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x20c8908 (0x7f662e6fc908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f662e0f70e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: <unknown function> + 0x3443e11 (0x7f662fa77e11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x34443bb (0x7f662fa783bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f662e150de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f662dc2b05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #20: <unknown function> + 0x2471e09 (0x7f662eaa5e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f662e2c23f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #22: <unknown function> + 0x36467f (0x7f663937767f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #23: <unknown function> + 0x364b3c (0x7f6639377b3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #24: <unknown function> + 0x1ddc68 (0x55b088505c68 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0x1049f3 (0x55b08842c9f3 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0x104425 (0x55b08842c425 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x18fc9b (0x55b0884b7c9b in /opt/conda/bin/python)\nframe #30: <unknown function> + 0x1052a5 (0x55b08842d2a5 in /opt/conda/bin/python)\nframe #31: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #32: <unknown function> + 0x18fc9b (0x55b0884b7c9b in /opt/conda/bin/python)\nframe #33: <unknown function> + 0x1052a5 (0x55b08842d2a5 in /opt/conda/bin/python)\nframe #34: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x20a (0x55b08846ffaa in /opt/conda/bin/python)\nframe #36: _PyEval_EvalFrameDefault + 0x26e4 (0x55b08850b774 in /opt/conda/bin/python)\nframe #37: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #38: _PyObject_Call + 0x20a (0x55b08846ffaa in /opt/conda/bin/python)\nframe #39: <unknown function> + 0xaa8dba (0x7f6639abbdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f6639ab9ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f6639abd2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f6639abd9a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f663105f5c4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f6639abd0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #45: <unknown function> + 0x4a249c3 (0x7f66310589c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f6631059558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f6631053857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #48: <unknown function> + 0x4a54542 (0x7f6631088542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f662134f90b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #50: <unknown function> + 0xdbbf4 (0x7f66511b5bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #51: <unknown function> + 0x76db (0x7f66718096db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #52: clone + 0x3f (0x7f667153261f in /lib/x86_64-linux-gnu/libc.so.6)\n\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 92, in _create_module\n    module.to(device)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1120, in to\n    return self._apply(convert)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 795, in _apply\n    param_applied = fn(param)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1118, in convert\n    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)\nRuntimeError: CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nException raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f662136159b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f662135cdfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7f66215e706e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #3: <unknown function> + 0x17c9d (0x7f66215bfc9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)\nframe #4: <unknown function> + 0xdf3ecd (0x7f66225ffecd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #5: <unknown function> + 0x29c7ab5 (0x7f66241d3ab5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #6: <unknown function> + 0x29c7c5b (0x7f66241d3c5b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f662e3d6cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #8: <unknown function> + 0x20c43b5 (0x7f662e6f83b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f662e411e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x12691af (0x7f662d89d1af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7f662dc33641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x22a6e23 (0x7f662e8dae23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f662e0f70e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #14: <unknown function> + 0x20c8908 (0x7f662e6fc908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f662e0f70e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #16: <unknown function> + 0x3443e11 (0x7f662fa77e11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #17: <unknown function> + 0x34443bb (0x7f662fa783bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f662e150de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f662dc2b05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #20: <unknown function> + 0x2471e09 (0x7f662eaa5e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f662e2c23f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #22: <unknown function> + 0x36467f (0x7f663937767f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #23: <unknown function> + 0x364b3c (0x7f6639377b3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #24: <unknown function> + 0x1ddc68 (0x55b088505c68 in /opt/conda/bin/python)\nframe #25: <unknown function> + 0x1049f3 (0x55b08842c9f3 in /opt/conda/bin/python)\nframe #26: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #27: <unknown function> + 0x104425 (0x55b08842c425 in /opt/conda/bin/python)\nframe #28: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #29: <unknown function> + 0x18fc9b (0x55b0884b7c9b in /opt/conda/bin/python)\nframe #30: <unknown function> + 0x1052a5 (0x55b08842d2a5 in /opt/conda/bin/python)\nframe #31: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #32: <unknown function> + 0x18fc9b (0x55b0884b7c9b in /opt/conda/bin/python)\nframe #33: <unknown function> + 0x1052a5 (0x55b08842d2a5 in /opt/conda/bin/python)\nframe #34: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #35: _PyObject_Call + 0x20a (0x55b08846ffaa in /opt/conda/bin/python)\nframe #36: _PyEval_EvalFrameDefault + 0x26e4 (0x55b08850b774 in /opt/conda/bin/python)\nframe #37: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)\nframe #38: _PyObject_Call + 0x20a (0x55b08846ffaa in /opt/conda/bin/python)\nframe #39: <unknown function> + 0xaa8dba (0x7f6639abbdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f6639ab9ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f6639abd2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f6639abd9a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f663105f5c4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f6639abd0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #45: <unknown function> + 0x4a249c3 (0x7f66310589c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f6631059558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f6631053857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #48: <unknown function> + 0x4a54542 (0x7f6631088542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f662134f90b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #50: <unknown function> + 0xdbbf4 (0x7f66511b5bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #51: <unknown function> + 0x76db (0x7f66718096db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #52: clone + 0x3f (0x7f667153261f in /lib/x86_64-linux-gnu/libc.so.6)\n\n')
2022-11-23T03:04:41.8394303Z Traceback (most recent call last):
2022-11-23T03:04:41.8394850Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T03:04:41.8395306Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T03:04:41.8395732Z   File "/tmp/tmp41k5ueqo/_remote_module_non_scriptable.py", line 47, in _remote_forward
2022-11-23T03:04:41.8396087Z     module = module_rref.local_value()
2022-11-23T03:04:41.8396616Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 236, in _handle_exception
2022-11-23T03:04:41.8396986Z     raise exc
2022-11-23T03:04:41.8397257Z RuntimeError: On WorkerInfo(id=1, name=worker1):
2022-11-23T03:04:41.8397657Z RuntimeError('CUDA error: invalid device ordinal
2022-11-23T03:04:41.8398104Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-11-23T03:04:41.8398535Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-11-23T03:04:41.8399029Z Exception raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):
2022-11-23T03:04:41.8399966Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f662136159b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.8400958Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f662135cdfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.8401837Z frame #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7f66215e706e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T03:04:41.8402504Z frame #3: <unknown function> + 0x17c9d (0x7f66215bfc9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T03:04:41.8403151Z frame #4: <unknown function> + 0xdf3ecd (0x7f66225ffecd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.8403795Z frame #5: <unknown function> + 0x29c7ab5 (0x7f66241d3ab5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.8404430Z frame #6: <unknown function> + 0x29c7c5b (0x7f66241d3c5b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.8405432Z frame #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f662e3d6cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8406276Z frame #8: <unknown function> + 0x20c43b5 (0x7f662e6f83b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8407311Z frame #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f662e411e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8408108Z frame #10: <unknown function> + 0x12691af (0x7f662d89d1af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8409057Z frame #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7f662dc33641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8409858Z frame #12: <unknown function> + 0x22a6e23 (0x7f662e8dae23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8410899Z frame #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f662e0f70e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8411747Z frame #14: <unknown function> + 0x20c8908 (0x7f662e6fc908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8412753Z frame #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f662e0f70e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8413593Z frame #16: <unknown function> + 0x3443e11 (0x7f662fa77e11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8414222Z frame #17: <unknown function> + 0x34443bb (0x7f662fa783bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8415222Z frame #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f662e150de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8416337Z frame #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f662dc2b05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8417135Z frame #20: <unknown function> + 0x2471e09 (0x7f662eaa5e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8418113Z frame #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f662e2c23f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8418934Z frame #22: <unknown function> + 0x36467f (0x7f663937767f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8419562Z frame #23: <unknown function> + 0x364b3c (0x7f6639377b3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8420017Z frame #24: <unknown function> + 0x1ddc68 (0x55b088505c68 in /opt/conda/bin/python)
2022-11-23T03:04:41.8420415Z frame #25: <unknown function> + 0x1049f3 (0x55b08842c9f3 in /opt/conda/bin/python)
2022-11-23T03:04:41.8420805Z frame #26: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8421180Z frame #27: <unknown function> + 0x104425 (0x55b08842c425 in /opt/conda/bin/python)
2022-11-23T03:04:41.8421633Z frame #28: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8422026Z frame #29: <unknown function> + 0x18fc9b (0x55b0884b7c9b in /opt/conda/bin/python)
2022-11-23T03:04:41.8422398Z frame #30: <unknown function> + 0x1052a5 (0x55b08842d2a5 in /opt/conda/bin/python)
2022-11-23T03:04:41.8422791Z frame #31: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8423180Z frame #32: <unknown function> + 0x18fc9b (0x55b0884b7c9b in /opt/conda/bin/python)
2022-11-23T03:04:41.8423570Z frame #33: <unknown function> + 0x1052a5 (0x55b08842d2a5 in /opt/conda/bin/python)
2022-11-23T03:04:41.8423934Z frame #34: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8424317Z frame #35: _PyObject_Call + 0x20a (0x55b08846ffaa in /opt/conda/bin/python)
2022-11-23T03:04:41.8424730Z frame #36: _PyEval_EvalFrameDefault + 0x26e4 (0x55b08850b774 in /opt/conda/bin/python)
2022-11-23T03:04:41.8425118Z frame #37: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8425508Z frame #38: _PyObject_Call + 0x20a (0x55b08846ffaa in /opt/conda/bin/python)
2022-11-23T03:04:41.8426112Z frame #39: <unknown function> + 0xaa8dba (0x7f6639abbdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8427096Z frame #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f6639ab9ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8428102Z frame #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f6639abd2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8429229Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f6639abd9a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8430535Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f663105f5c4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8431817Z frame #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f6639abd0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8432695Z frame #45: <unknown function> + 0x4a249c3 (0x7f66310589c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8433622Z frame #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f6631059558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8434684Z frame #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f6631053857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8435437Z frame #48: <unknown function> + 0x4a54542 (0x7f6631088542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8436256Z frame #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f662134f90b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.8436891Z frame #50: <unknown function> + 0xdbbf4 (0x7f66511b5bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T03:04:41.8437519Z frame #51: <unknown function> + 0x76db (0x7f66718096db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T03:04:41.8438110Z frame #52: clone + 0x3f (0x7f667153261f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T03:04:41.8438687Z ')
2022-11-23T03:04:41.8439010Z Traceback (most recent call last):
2022-11-23T03:04:41.8439594Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T03:04:41.8440127Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T03:04:41.8440838Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 92, in _create_module
2022-11-23T03:04:41.8441355Z     module.to(device)
2022-11-23T03:04:41.8473310Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1120, in to
2022-11-23T03:04:41.8473766Z     return self._apply(convert)
2022-11-23T03:04:41.8474299Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 795, in _apply
2022-11-23T03:04:41.8474676Z     param_applied = fn(param)
2022-11-23T03:04:41.8475157Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1118, in convert
2022-11-23T03:04:41.8475630Z     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2022-11-23T03:04:41.8476009Z RuntimeError: CUDA error: invalid device ordinal
2022-11-23T03:04:41.8476467Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-11-23T03:04:41.8476918Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-11-23T03:04:41.8477405Z Exception raised from c10_cuda_check_implementation at /var/lib/jenkins/workspace/c10/cuda/CUDAException.cpp:31 (most recent call first):
2022-11-23T03:04:41.8478259Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f662136159b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.8479237Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f662135cdfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.8480275Z frame #2: c10::cuda::c10_cuda_check_implementation(char const*, char const*, int, bool) + 0x42e (0x7f66215e706e in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T03:04:41.8480995Z frame #3: <unknown function> + 0x17c9d (0x7f66215bfc9d in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
2022-11-23T03:04:41.8481632Z frame #4: <unknown function> + 0xdf3ecd (0x7f66225ffecd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.8482260Z frame #5: <unknown function> + 0x29c7ab5 (0x7f66241d3ab5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.8482896Z frame #6: <unknown function> + 0x29c7c5b (0x7f66241d3c5b in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.8483925Z frame #7: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x1e3 (0x7f662e3d6cb3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8484767Z frame #8: <unknown function> + 0x20c43b5 (0x7f662e6f83b5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8485696Z frame #9: at::_ops::empty_strided::call(c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>) + 0x168 (0x7f662e411e58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8486500Z frame #10: <unknown function> + 0x12691af (0x7f662d89d1af in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8487541Z frame #11: at::native::_to_copy(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x1321 (0x7f662dc33641 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8488341Z frame #12: <unknown function> + 0x22a6e23 (0x7f662e8dae23 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8489348Z frame #13: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f662e0f70e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8490189Z frame #14: <unknown function> + 0x20c8908 (0x7f662e6fc908 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8491192Z frame #15: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x103 (0x7f662e0f70e3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8492031Z frame #16: <unknown function> + 0x3443e11 (0x7f662fa77e11 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8492667Z frame #17: <unknown function> + 0x34443bb (0x7f662fa783bb in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8493615Z frame #18: at::_ops::_to_copy::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, c10::optional<c10::MemoryFormat>) + 0x201 (0x7f662e150de1 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8494723Z frame #19: at::native::to(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x13e (0x7f662dc2b05e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8495562Z frame #20: <unknown function> + 0x2471e09 (0x7f662eaa5e09 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8496553Z frame #21: at::_ops::to_dtype_layout::call(at::Tensor const&, c10::optional<c10::ScalarType>, c10::optional<c10::Layout>, c10::optional<c10::Device>, c10::optional<bool>, bool, bool, c10::optional<c10::MemoryFormat>) + 0x215 (0x7f662e2c23f5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8497382Z frame #22: <unknown function> + 0x36467f (0x7f663937767f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8498033Z frame #23: <unknown function> + 0x364b3c (0x7f6639377b3c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8498490Z frame #24: <unknown function> + 0x1ddc68 (0x55b088505c68 in /opt/conda/bin/python)
2022-11-23T03:04:41.8498869Z frame #25: <unknown function> + 0x1049f3 (0x55b08842c9f3 in /opt/conda/bin/python)
2022-11-23T03:04:41.8499261Z frame #26: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8499647Z frame #27: <unknown function> + 0x104425 (0x55b08842c425 in /opt/conda/bin/python)
2022-11-23T03:04:41.8500014Z frame #28: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8500403Z frame #29: <unknown function> + 0x18fc9b (0x55b0884b7c9b in /opt/conda/bin/python)
2022-11-23T03:04:41.8500791Z frame #30: <unknown function> + 0x1052a5 (0x55b08842d2a5 in /opt/conda/bin/python)
2022-11-23T03:04:41.8501174Z frame #31: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8501603Z frame #32: <unknown function> + 0x18fc9b (0x55b0884b7c9b in /opt/conda/bin/python)
2022-11-23T03:04:41.8501985Z frame #33: <unknown function> + 0x1052a5 (0x55b08842d2a5 in /opt/conda/bin/python)
2022-11-23T03:04:41.8502374Z frame #34: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8502745Z frame #35: _PyObject_Call + 0x20a (0x55b08846ffaa in /opt/conda/bin/python)
2022-11-23T03:04:41.8503158Z frame #36: _PyEval_EvalFrameDefault + 0x26e4 (0x55b08850b774 in /opt/conda/bin/python)
2022-11-23T03:04:41.8503563Z frame #37: <unknown function> + 0x18f742 (0x55b0884b7742 in /opt/conda/bin/python)
2022-11-23T03:04:41.8503944Z frame #38: _PyObject_Call + 0x20a (0x55b08846ffaa in /opt/conda/bin/python)
2022-11-23T03:04:41.8504529Z frame #39: <unknown function> + 0xaa8dba (0x7f6639abbdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8505323Z frame #40: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f6639ab9ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8506343Z frame #41: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f6639abd2d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8507705Z frame #42: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x83 (0x7f6639abd9a3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8508926Z frame #43: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f663105f5c4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8510295Z frame #44: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f6639abd0c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.8511226Z frame #45: <unknown function> + 0x4a249c3 (0x7f66310589c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8512148Z frame #46: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f6631059558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8513199Z frame #47: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f6631053857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8513977Z frame #48: <unknown function> + 0x4a54542 (0x7f6631088542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.8514655Z frame #49: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f662134f90b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.8515144Z frame #50: <unknown function> + 0xdbbf4 (0x7f66511b5bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T03:04:41.8515692Z frame #51: <unknown function> + 0x76db (0x7f66718096db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T03:04:41.8516184Z frame #52: clone + 0x3f (0x7f667153261f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T03:04:41.8516405Z 
2022-11-23T03:04:41.8516423Z 
2022-11-23T03:04:41.8516441Z 
2022-11-23T03:04:41.8516542Z ok (4.628s)
2022-11-23T03:04:41.8516669Z 
2022-11-23T03:04:41.8516937Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8517345Z Ran 1 test in 4.628s
2022-11-23T03:04:41.8517506Z 
2022-11-23T03:04:41.8517597Z OK
2022-11-23T03:04:41.8517731Z 
2022-11-23T03:04:41.8517837Z Generating XML reports...
2022-11-23T03:04:41.8518520Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024458.xml
2022-11-23T03:04:41.8519288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8519739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8520293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8520760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8521223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8e66r1su
2022-11-23T03:04:41.8521751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8e66r1su/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8522055Z 
2022-11-23T03:04:41.8522163Z Running tests...
2022-11-23T03:04:41.8522567Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8523142Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8523675Z   test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8524172Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77371
2022-11-23T03:04:41.8524624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77372
2022-11-23T03:04:41.8525220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8525667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8526247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8526712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8527349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8527805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8528377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8528840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8529287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpec4tlz1y
2022-11-23T03:04:41.8529824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpec4tlz1y/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8530358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz_nvnz5
2022-11-23T03:04:41.8530872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz_nvnz5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8531377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8531851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8532241Z fi_getinfo: -61
2022-11-23T03:04:41.8532497Z fi_getinfo: -61
2022-11-23T03:04:41.8532726Z ok (6.025s)
2022-11-23T03:04:41.8532874Z 
2022-11-23T03:04:41.8533136Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8533444Z Ran 1 test in 6.025s
2022-11-23T03:04:41.8533602Z 
2022-11-23T03:04:41.8533697Z OK
2022-11-23T03:04:41.8533830Z 
2022-11-23T03:04:41.8533952Z Generating XML reports...
2022-11-23T03:04:41.8534605Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024506.xml
2022-11-23T03:04:41.8535437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8535885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8536463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8536911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8537373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekbksa9q
2022-11-23T03:04:41.8537915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekbksa9q/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8538218Z 
2022-11-23T03:04:41.8538325Z Running tests...
2022-11-23T03:04:41.8538709Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8539286Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8539816Z   test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8540279Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77561
2022-11-23T03:04:41.8540732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77562
2022-11-23T03:04:41.8541174Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77563
2022-11-23T03:04:41.8541617Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77564
2022-11-23T03:04:41.8542212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8542660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8543281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8544042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8544638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8545076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8545717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8546176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8547045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8547527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8548059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8548524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8549094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8549527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8550078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8550536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8550999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvfal1ils
2022-11-23T03:04:41.8551532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvfal1ils/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8552050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmgh1k4f
2022-11-23T03:04:41.8552586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmgh1k4f/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8553230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpjq67ln
2022-11-23T03:04:41.8553749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpjq67ln/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8554284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6mll6fd
2022-11-23T03:04:41.8554815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6mll6fd/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8555317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8555771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8556234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8556702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8557085Z fi_getinfo: -61
2022-11-23T03:04:41.8557361Z fi_getinfo: -61
2022-11-23T03:04:41.8557626Z fi_getinfo: -61
2022-11-23T03:04:41.8557879Z fi_getinfo: -61
2022-11-23T03:04:41.8558108Z ok (8.946s)
2022-11-23T03:04:41.8558254Z 
2022-11-23T03:04:41.8558519Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8558846Z Ran 1 test in 8.946s
2022-11-23T03:04:41.8558989Z 
2022-11-23T03:04:41.8559082Z OK
2022-11-23T03:04:41.8559214Z 
2022-11-23T03:04:41.8559337Z Generating XML reports...
2022-11-23T03:04:41.8559973Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20221123024514.xml
2022-11-23T03:04:41.8560696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8561145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8561718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8562186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8562632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnci1ht7
2022-11-23T03:04:41.8563233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnci1ht7/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8563543Z 
2022-11-23T03:04:41.8563652Z Running tests...
2022-11-23T03:04:41.8564041Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8564614Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8565163Z   test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8565661Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77908
2022-11-23T03:04:41.8566098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77909
2022-11-23T03:04:41.8566699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8567147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8567723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8568174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8568748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8569187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8569734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8570192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8570721Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpww1bdqgg
2022-11-23T03:04:41.8571261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpww1bdqgg/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8571781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_isxgmvz
2022-11-23T03:04:41.8572312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_isxgmvz/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8572817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8573271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8573655Z skip: Need at least 4 CUDA devices (4.210s)
2022-11-23T03:04:41.8573847Z 
2022-11-23T03:04:41.8574120Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8574449Z Ran 1 test in 4.211s
2022-11-23T03:04:41.8574590Z 
2022-11-23T03:04:41.8574698Z OK (skipped=1)
2022-11-23T03:04:41.8574849Z 
2022-11-23T03:04:41.8574971Z Generating XML reports...
2022-11-23T03:04:41.8575622Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024526.xml
2022-11-23T03:04:41.8576360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8576808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8577376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8577839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8578285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_z2p_u7
2022-11-23T03:04:41.8578821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_z2p_u7/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8579123Z 
2022-11-23T03:04:41.8579230Z Running tests...
2022-11-23T03:04:41.8579628Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8580186Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8580786Z   test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8581296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78011
2022-11-23T03:04:41.8581730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78012
2022-11-23T03:04:41.8582329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8582777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8583345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8583802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8584373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8584815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8585368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8585829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8586291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcf06nqhr
2022-11-23T03:04:41.8587003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcf06nqhr/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8587525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4l_qt4r5
2022-11-23T03:04:41.8588147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4l_qt4r5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8588651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8589127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8589500Z skip: Need at least 4 CUDA devices (4.330s)
2022-11-23T03:04:41.8589691Z 
2022-11-23T03:04:41.8589970Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8590294Z Ran 1 test in 4.330s
2022-11-23T03:04:41.8590452Z 
2022-11-23T03:04:41.8590542Z OK (skipped=1)
2022-11-23T03:04:41.8590694Z 
2022-11-23T03:04:41.8590817Z Generating XML reports...
2022-11-23T03:04:41.8591468Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024533.xml
2022-11-23T03:04:41.8592224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8592651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8593222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8593690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8594137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vsx63j7
2022-11-23T03:04:41.8594668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vsx63j7/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8594965Z 
2022-11-23T03:04:41.8595071Z Running tests...
2022-11-23T03:04:41.8595472Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8596028Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8596579Z   test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8597073Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78114
2022-11-23T03:04:41.8597507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78115
2022-11-23T03:04:41.8598185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8598644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8599220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8599674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8600251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8600694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8601267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8601767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8602236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptokz8hyk
2022-11-23T03:04:41.8602776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptokz8hyk/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8603290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyls93r9t
2022-11-23T03:04:41.8603822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyls93r9t/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8604323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8604791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8605229Z skip: Need at least 4 CUDA devices (4.237s)
2022-11-23T03:04:41.8605419Z 
2022-11-23T03:04:41.8605690Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8606014Z Ran 1 test in 4.237s
2022-11-23T03:04:41.8606172Z 
2022-11-23T03:04:41.8606262Z OK (skipped=1)
2022-11-23T03:04:41.8606413Z 
2022-11-23T03:04:41.8606538Z Generating XML reports...
2022-11-23T03:04:41.8607188Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024540.xml
2022-11-23T03:04:41.8607931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8608361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8608928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8609400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8609845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3deefx9_
2022-11-23T03:04:41.8610375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3deefx9_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8610711Z 
2022-11-23T03:04:41.8610816Z Running tests...
2022-11-23T03:04:41.8611218Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8611771Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8612320Z   test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8612815Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78217
2022-11-23T03:04:41.8613263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78218
2022-11-23T03:04:41.8613851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8614294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8614860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8615373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8615956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8616394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8616960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8617399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8617858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9cjt1eh
2022-11-23T03:04:41.8618401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9cjt1eh/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8618920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxlh7k6ct
2022-11-23T03:04:41.8619454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxlh7k6ct/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8619960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8620430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8620803Z skip: Need at least 4 CUDA devices (4.367s)
2022-11-23T03:04:41.8620995Z 
2022-11-23T03:04:41.8621267Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8621588Z Ran 1 test in 4.367s
2022-11-23T03:04:41.8621747Z 
2022-11-23T03:04:41.8621854Z OK (skipped=1)
2022-11-23T03:04:41.8622051Z 
2022-11-23T03:04:41.8622174Z Generating XML reports...
2022-11-23T03:04:41.8622830Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024547.xml
2022-11-23T03:04:41.8623579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8624017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8624593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8625058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8625521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmploeod57c
2022-11-23T03:04:41.8626043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmploeod57c/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8626343Z 
2022-11-23T03:04:41.8626456Z Running tests...
2022-11-23T03:04:41.8627035Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8627612Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8628152Z   test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8628652Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78320
2022-11-23T03:04:41.8629106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78321
2022-11-23T03:04:41.8629696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8630139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8630708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8631165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8631738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8632176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8632828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8633290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8633751Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfirgrn4k
2022-11-23T03:04:41.8634294Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfirgrn4k/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8634827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptp1lem01
2022-11-23T03:04:41.8635342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptp1lem01/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8635852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8636321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8636695Z skip: Need at least 4 CUDA devices (4.245s)
2022-11-23T03:04:41.8636888Z 
2022-11-23T03:04:41.8637166Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8637496Z Ran 1 test in 4.245s
2022-11-23T03:04:41.8637655Z 
2022-11-23T03:04:41.8637761Z OK (skipped=1)
2022-11-23T03:04:41.8637897Z 
2022-11-23T03:04:41.8638019Z Generating XML reports...
2022-11-23T03:04:41.8638671Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024554.xml
2022-11-23T03:04:41.8639415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8639923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8640499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8640963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8641429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcoxat5_
2022-11-23T03:04:41.8641951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcoxat5_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8642250Z 
2022-11-23T03:04:41.8642356Z Running tests...
2022-11-23T03:04:41.8642758Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8643312Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8643870Z   test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8644381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78423
2022-11-23T03:04:41.8644831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78424
2022-11-23T03:04:41.8645420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8645870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8646445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8646915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8647471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8647915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8648479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8648926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8649389Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsa6fqvlp
2022-11-23T03:04:41.8649999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsa6fqvlp/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8650540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppym3019b
2022-11-23T03:04:41.8651055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppym3019b/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8651558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8652027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8652401Z skip: Need at least 4 CUDA devices (4.267s)
2022-11-23T03:04:41.8652599Z 
2022-11-23T03:04:41.8652875Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8653199Z Ran 1 test in 4.267s
2022-11-23T03:04:41.8653357Z 
2022-11-23T03:04:41.8653464Z OK (skipped=1)
2022-11-23T03:04:41.8653600Z 
2022-11-23T03:04:41.8653722Z Generating XML reports...
2022-11-23T03:04:41.8654376Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024601.xml
2022-11-23T03:04:41.8655130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8655579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8656133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8656598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8657060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1sckuud0
2022-11-23T03:04:41.8657651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1sckuud0/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8657951Z 
2022-11-23T03:04:41.8658058Z Running tests...
2022-11-23T03:04:41.8658462Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8659034Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8659569Z   test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8660060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78526
2022-11-23T03:04:41.8660510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78527
2022-11-23T03:04:41.8661102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8661553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8662123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8662588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8663150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8663593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8664160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8664604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8665066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm12ssln4
2022-11-23T03:04:41.8665599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm12ssln4/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8666134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqra0mib
2022-11-23T03:04:41.8666653Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqra0mib/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8667420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8667905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8668295Z skip: Need at least 4 CUDA devices (4.241s)
2022-11-23T03:04:41.8668471Z 
2022-11-23T03:04:41.8668748Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8669075Z Ran 1 test in 4.241s
2022-11-23T03:04:41.8669235Z 
2022-11-23T03:04:41.8669341Z OK (skipped=1)
2022-11-23T03:04:41.8669494Z 
2022-11-23T03:04:41.8669598Z Generating XML reports...
2022-11-23T03:04:41.8670251Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024608.xml
2022-11-23T03:04:41.8671003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8671452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8672013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8672479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8672941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90xnro48
2022-11-23T03:04:41.8673454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90xnro48/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8673754Z 
2022-11-23T03:04:41.8673866Z Running tests...
2022-11-23T03:04:41.8674265Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8674920Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8675465Z   test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8675972Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78629
2022-11-23T03:04:41.8676425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78630
2022-11-23T03:04:41.8677018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8677463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8678033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8678495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8679048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8679499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8680065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8680527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8680974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxf8rw26v
2022-11-23T03:04:41.8681513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxf8rw26v/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8682046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeucu6kj4
2022-11-23T03:04:41.8682564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeucu6kj4/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8683074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8683547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8683936Z skip: Need at least 4 CUDA devices (4.307s)
2022-11-23T03:04:41.8684108Z 
2022-11-23T03:04:41.8684382Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8684760Z Ran 1 test in 4.307s
2022-11-23T03:04:41.8684926Z 
2022-11-23T03:04:41.8685033Z OK (skipped=1)
2022-11-23T03:04:41.8685186Z 
2022-11-23T03:04:41.8685292Z Generating XML reports...
2022-11-23T03:04:41.8685949Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024615.xml
2022-11-23T03:04:41.8686698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8687148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8687704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8688172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8688635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrku0huz
2022-11-23T03:04:41.8689180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrku0huz/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8689465Z 
2022-11-23T03:04:41.8689573Z Running tests...
2022-11-23T03:04:41.8689973Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8690540Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8691129Z   test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8691672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78732
2022-11-23T03:04:41.8692192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78733
2022-11-23T03:04:41.8692641Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 78734
2022-11-23T03:04:41.8693068Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 78735
2022-11-23T03:04:41.8693685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8694136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8694693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8695159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8695734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8696180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8696735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8697196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8697781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8698221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8698773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8699235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8699806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8700224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8700795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8701257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8701719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph6bv18vl
2022-11-23T03:04:41.8702292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph6bv18vl/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8702830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0cxcgyc
2022-11-23T03:04:41.8703363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0cxcgyc/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8703873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82osuxj_
2022-11-23T03:04:41.8704404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82osuxj_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8704932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5qbvfzak
2022-11-23T03:04:41.8705465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5qbvfzak/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8705955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8706428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8707169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8707647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8708033Z fi_getinfo: -61
2022-11-23T03:04:41.8708307Z fi_getinfo: -61
2022-11-23T03:04:41.8708576Z fi_getinfo: -61
2022-11-23T03:04:41.8708828Z fi_getinfo: -61
2022-11-23T03:04:41.8709055Z ok (11.968s)
2022-11-23T03:04:41.8709201Z 
2022-11-23T03:04:41.8709466Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8709930Z Ran 1 test in 11.968s
2022-11-23T03:04:41.8710089Z 
2022-11-23T03:04:41.8710181Z OK
2022-11-23T03:04:41.8710201Z 
2022-11-23T03:04:41.8710323Z Generating XML reports...
2022-11-23T03:04:41.8710922Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024622.xml
2022-11-23T03:04:41.8711296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8711471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8711831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8712018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8712269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpypt4z_da
2022-11-23T03:04:41.8712542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpypt4z_da/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8712562Z 
2022-11-23T03:04:41.8712667Z Running tests...
2022-11-23T03:04:41.8712929Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8713290Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8713609Z   test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8713830Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79079
2022-11-23T03:04:41.8714029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79080
2022-11-23T03:04:41.8714243Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79081
2022-11-23T03:04:41.8714454Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79082
2022-11-23T03:04:41.8714829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8715005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8715449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8715646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8716011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8716165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8716537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8716728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8717089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8717261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8717634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8717824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8718180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8718353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8718708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8718896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8719150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpjrbcep
2022-11-23T03:04:41.8719479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpjrbcep/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8719738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpymntprw1
2022-11-23T03:04:41.8720012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpymntprw1/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8720264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ws_54ff
2022-11-23T03:04:41.8720528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ws_54ff/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8720761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjeerwapx
2022-11-23T03:04:41.8721024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjeerwapx/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8721251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8721481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8721707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8721932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8722088Z fi_getinfo: -61
2022-11-23T03:04:41.8722225Z fi_getinfo: -61
2022-11-23T03:04:41.8722341Z fi_getinfo: -61
2022-11-23T03:04:41.8722477Z fi_getinfo: -61
2022-11-23T03:04:41.8722575Z ok (11.867s)
2022-11-23T03:04:41.8722595Z 
2022-11-23T03:04:41.8722859Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8722971Z Ran 1 test in 11.868s
2022-11-23T03:04:41.8722990Z 
2022-11-23T03:04:41.8723085Z OK
2022-11-23T03:04:41.8723104Z 
2022-11-23T03:04:41.8723228Z Generating XML reports...
2022-11-23T03:04:41.8723774Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024636.xml
2022-11-23T03:04:41.8724134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8724308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8724732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8724930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8725184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsi8_3js_
2022-11-23T03:04:41.8725450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsi8_3js_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8725470Z 
2022-11-23T03:04:41.8725577Z Running tests...
2022-11-23T03:04:41.8725841Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8726186Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8726512Z   test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8726731Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79426
2022-11-23T03:04:41.8726951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79427
2022-11-23T03:04:41.8727164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79428
2022-11-23T03:04:41.8727374Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79429
2022-11-23T03:04:41.8727743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8727918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8728295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8728524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8728887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8729062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8729434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8729624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8729981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8730153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8730520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8730711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8731060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8731233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8731609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8731796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8732049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpofd3t01b
2022-11-23T03:04:41.8732319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpofd3t01b/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8732568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpne_eof5m
2022-11-23T03:04:41.8732838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpne_eof5m/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8733072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfz67jfq4
2022-11-23T03:04:41.8733334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfz67jfq4/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8733632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprg6qb6ne
2022-11-23T03:04:41.8733903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprg6qb6ne/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8734132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8734360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8734586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8734807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8734912Z ok (11.325s)
2022-11-23T03:04:41.8734932Z 
2022-11-23T03:04:41.8735185Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8735297Z Ran 1 test in 11.325s
2022-11-23T03:04:41.8735316Z 
2022-11-23T03:04:41.8735406Z OK
2022-11-23T03:04:41.8735426Z 
2022-11-23T03:04:41.8735550Z Generating XML reports...
2022-11-23T03:04:41.8736097Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024651.xml
2022-11-23T03:04:41.8736465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8736638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8737013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8737242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8737495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8sg77z2h
2022-11-23T03:04:41.8737760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8sg77z2h/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8737779Z 
2022-11-23T03:04:41.8737890Z Running tests...
2022-11-23T03:04:41.8738162Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8738522Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8738857Z   test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8739075Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79605
2022-11-23T03:04:41.8739291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79606
2022-11-23T03:04:41.8739493Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79607
2022-11-23T03:04:41.8739704Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79608
2022-11-23T03:04:41.8740073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8740250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8740633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8740823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8741181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8741351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8741703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8741897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8742251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8742485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8742865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8743050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8743414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8743585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8743952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8744126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8744383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2ktaak4
2022-11-23T03:04:41.8744652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2ktaak4/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8744907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcqtulop8
2022-11-23T03:04:41.8745174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcqtulop8/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8745425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmlmsxg6
2022-11-23T03:04:41.8745694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmlmsxg6/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8745942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj35seo8t
2022-11-23T03:04:41.8746249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj35seo8t/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8746477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8746704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8747138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8747368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8747472Z ok (9.966s)
2022-11-23T03:04:41.8747496Z 
2022-11-23T03:04:41.8747766Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8747877Z Ran 1 test in 9.966s
2022-11-23T03:04:41.8747896Z 
2022-11-23T03:04:41.8747989Z OK
2022-11-23T03:04:41.8748008Z 
2022-11-23T03:04:41.8748113Z Generating XML reports...
2022-11-23T03:04:41.8748662Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024705.xml
2022-11-23T03:04:41.8749037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8749212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8749593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8749783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8750036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpst7y3w_r
2022-11-23T03:04:41.8750304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpst7y3w_r/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8750323Z 
2022-11-23T03:04:41.8750431Z Running tests...
2022-11-23T03:04:41.8750676Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8751037Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8751357Z   test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8751666Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79840
2022-11-23T03:04:41.8751894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79841
2022-11-23T03:04:41.8752105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 79842
2022-11-23T03:04:41.8752315Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 79843
2022-11-23T03:04:41.8752689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8752847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8753229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8753416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8753773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8753947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8754320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8754508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8754862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8755036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8755391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8755648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8756008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8756181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8756549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8756733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8756988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7w1dxgo
2022-11-23T03:04:41.8757258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7w1dxgo/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8757497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphzrgkq5i
2022-11-23T03:04:41.8757771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphzrgkq5i/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8758021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88d4dqw7
2022-11-23T03:04:41.8758285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88d4dqw7/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8758536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpihf3z5z0
2022-11-23T03:04:41.8758797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpihf3z5z0/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8759024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8759251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8759477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8759690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8759790Z ok (9.814s)
2022-11-23T03:04:41.8759810Z 
2022-11-23T03:04:41.8760080Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8760191Z Ran 1 test in 9.814s
2022-11-23T03:04:41.8760210Z 
2022-11-23T03:04:41.8760303Z OK
2022-11-23T03:04:41.8760370Z 
2022-11-23T03:04:41.8760499Z Generating XML reports...
2022-11-23T03:04:41.8761051Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024718.xml
2022-11-23T03:04:41.8761422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8761580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8761958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8762153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8762405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpya56z8zk
2022-11-23T03:04:41.8762670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpya56z8zk/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8762689Z 
2022-11-23T03:04:41.8762798Z Running tests...
2022-11-23T03:04:41.8763061Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8763414Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8763767Z   test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8763968Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80015
2022-11-23T03:04:41.8764183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80016
2022-11-23T03:04:41.8764453Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80017
2022-11-23T03:04:41.8764663Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80018
2022-11-23T03:04:41.8765043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8765216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8765595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8765785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8766129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8766299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8766674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8766862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8767217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8767394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8767762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8767949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8768312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8768466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8768832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8769020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8769273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5myrujic
2022-11-23T03:04:41.8769590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5myrujic/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8769846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79g_cza0
2022-11-23T03:04:41.8770110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79g_cza0/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8770360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0s556gt3
2022-11-23T03:04:41.8770624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0s556gt3/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8770860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsuriku2l
2022-11-23T03:04:41.8771129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsuriku2l/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8771356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8771584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8771811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8772034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8772135Z ok (9.941s)
2022-11-23T03:04:41.8772155Z 
2022-11-23T03:04:41.8772425Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8772519Z Ran 1 test in 9.941s
2022-11-23T03:04:41.8772537Z 
2022-11-23T03:04:41.8772627Z OK
2022-11-23T03:04:41.8772646Z 
2022-11-23T03:04:41.8772768Z Generating XML reports...
2022-11-23T03:04:41.8773370Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024730.xml
2022-11-23T03:04:41.8773736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8773915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8774295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8774483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8774740Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgh4d8di
2022-11-23T03:04:41.8774991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgh4d8di/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8775011Z 
2022-11-23T03:04:41.8775117Z Running tests...
2022-11-23T03:04:41.8775383Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8775741Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8776083Z   test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8776305Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80254
2022-11-23T03:04:41.8776521Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80255
2022-11-23T03:04:41.8776735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80256
2022-11-23T03:04:41.8776928Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80257
2022-11-23T03:04:41.8777305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8777477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8777862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8778050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8778457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8778634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8779006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8779195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8779534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8779702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8780073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8780258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8780622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8780796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8781162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8781348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8781586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgp7czsts
2022-11-23T03:04:41.8781856Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgp7czsts/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8782110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnyaxobs3
2022-11-23T03:04:41.8782436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnyaxobs3/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8782688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjn8logr_
2022-11-23T03:04:41.8782956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjn8logr_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8783207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucsp5f8k
2022-11-23T03:04:41.8783471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucsp5f8k/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8783697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8783908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8784133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8784359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8784461Z ok (9.843s)
2022-11-23T03:04:41.8784480Z 
2022-11-23T03:04:41.8784746Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8784855Z Ran 1 test in 9.843s
2022-11-23T03:04:41.8784878Z 
2022-11-23T03:04:41.8784969Z OK
2022-11-23T03:04:41.8784989Z 
2022-11-23T03:04:41.8785111Z Generating XML reports...
2022-11-23T03:04:41.8785641Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024743.xml
2022-11-23T03:04:41.8786010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8786186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8786561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8786753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8787181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpguawm60a
2022-11-23T03:04:41.8787530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpguawm60a/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8787545Z 
2022-11-23T03:04:41.8787662Z Running tests...
2022-11-23T03:04:41.8787931Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8788271Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8788613Z   test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8788830Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80433
2022-11-23T03:04:41.8789052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80434
2022-11-23T03:04:41.8789266Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80435
2022-11-23T03:04:41.8789475Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80436
2022-11-23T03:04:41.8789854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8790029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8790406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8790580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8790940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8791109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8791549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8791738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8792101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8792277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8792652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8792822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8793182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8793354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8793730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8793917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8794173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmu6znlg5
2022-11-23T03:04:41.8794448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmu6znlg5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8794702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp300cn2c9
2022-11-23T03:04:41.8794964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp300cn2c9/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8795198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4xx4kss
2022-11-23T03:04:41.8795461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4xx4kss/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8795719Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphe25gp07
2022-11-23T03:04:41.8795985Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphe25gp07/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8796214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8796488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8796722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8796946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8797029Z ok (9.948s)
2022-11-23T03:04:41.8797065Z 
2022-11-23T03:04:41.8797317Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8797426Z Ran 1 test in 9.949s
2022-11-23T03:04:41.8797445Z 
2022-11-23T03:04:41.8797538Z OK
2022-11-23T03:04:41.8797557Z 
2022-11-23T03:04:41.8797682Z Generating XML reports...
2022-11-23T03:04:41.8798228Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024755.xml
2022-11-23T03:04:41.8798596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8798775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8799155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8799328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8799581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7go2aree
2022-11-23T03:04:41.8799849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7go2aree/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8799869Z 
2022-11-23T03:04:41.8800029Z Running tests...
2022-11-23T03:04:41.8800292Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8800649Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8800985Z   test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8801202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80668
2022-11-23T03:04:41.8801401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80669
2022-11-23T03:04:41.8801619Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80670
2022-11-23T03:04:41.8801831Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80671
2022-11-23T03:04:41.8802202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8802381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8802758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8802947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8803308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8803481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8803835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8804020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8804385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8804555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8804926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8805111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8805536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8805714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8806069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8806255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8806509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp79jq29gh
2022-11-23T03:04:41.8806777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp79jq29gh/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8807034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps83ozutx
2022-11-23T03:04:41.8807301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps83ozutx/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8807551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxb1yvxm1
2022-11-23T03:04:41.8807819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxb1yvxm1/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8808068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5c3vz_h5
2022-11-23T03:04:41.8808315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5c3vz_h5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8808543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8808768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8809048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8809271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8809372Z ok (9.826s)
2022-11-23T03:04:41.8809392Z 
2022-11-23T03:04:41.8809658Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8809773Z Ran 1 test in 9.826s
2022-11-23T03:04:41.8809793Z 
2022-11-23T03:04:41.8809868Z OK
2022-11-23T03:04:41.8809903Z 
2022-11-23T03:04:41.8810008Z Generating XML reports...
2022-11-23T03:04:41.8810558Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024808.xml
2022-11-23T03:04:41.8810968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8811142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8811526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8811714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8811966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiwr7s7_m
2022-11-23T03:04:41.8812236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiwr7s7_m/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8812255Z 
2022-11-23T03:04:41.8812346Z Running tests...
2022-11-23T03:04:41.8812606Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8812961Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8813271Z   test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8813488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80843
2022-11-23T03:04:41.8813709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80844
2022-11-23T03:04:41.8813924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80845
2022-11-23T03:04:41.8814134Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80846
2022-11-23T03:04:41.8814558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8814721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8815099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8815289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8815649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8815824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8816193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8816382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8816751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8816904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8817269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8817455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8817817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8817986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8818410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8818595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8818846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4egw_68l
2022-11-23T03:04:41.8819114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4egw_68l/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8819350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgafny5i4
2022-11-23T03:04:41.8819617Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgafny5i4/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8819870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp03kss8ek
2022-11-23T03:04:41.8820133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp03kss8ek/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8820386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo1nzfmm7
2022-11-23T03:04:41.8820646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo1nzfmm7/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8820873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8821104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8821313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8821539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8821640Z ok (4.426s)
2022-11-23T03:04:41.8821659Z 
2022-11-23T03:04:41.8821925Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8822035Z Ran 1 test in 4.426s
2022-11-23T03:04:41.8822054Z 
2022-11-23T03:04:41.8822147Z OK
2022-11-23T03:04:41.8822169Z 
2022-11-23T03:04:41.8822291Z Generating XML reports...
2022-11-23T03:04:41.8822837Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024820.xml
2022-11-23T03:04:41.8823206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8823414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8823804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8823993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8824242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75xf0j47
2022-11-23T03:04:41.8824506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75xf0j47/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8824530Z 
2022-11-23T03:04:41.8824636Z Running tests...
2022-11-23T03:04:41.8824895Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8825253Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8825546Z   test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8825765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81014
2022-11-23T03:04:41.8825980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81015
2022-11-23T03:04:41.8826194Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81016
2022-11-23T03:04:41.8826406Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81017
2022-11-23T03:04:41.8826939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8827211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8827601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8827791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8828139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8828310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8828684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8828873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8829230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8829399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8829780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8829968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8830320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8830495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8830866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8831052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8831308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ha36z1y
2022-11-23T03:04:41.8831579Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ha36z1y/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8831837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvb164d92
2022-11-23T03:04:41.8832099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvb164d92/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8832349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2u7zpj1z
2022-11-23T03:04:41.8832664Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2u7zpj1z/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8832925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnhf9ryd
2022-11-23T03:04:41.8833191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnhf9ryd/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8833417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8833643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8833874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8834098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8834197Z ok (4.442s)
2022-11-23T03:04:41.8834217Z 
2022-11-23T03:04:41.8834485Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8834584Z Ran 1 test in 4.442s
2022-11-23T03:04:41.8834603Z 
2022-11-23T03:04:41.8834695Z OK
2022-11-23T03:04:41.8834714Z 
2022-11-23T03:04:41.8834835Z Generating XML reports...
2022-11-23T03:04:41.8835380Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024827.xml
2022-11-23T03:04:41.8835747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8835922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8836362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8836551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8836785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvwfuvu5
2022-11-23T03:04:41.8837056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvwfuvu5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8837076Z 
2022-11-23T03:04:41.8837183Z Running tests...
2022-11-23T03:04:41.8837447Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8837805Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8838112Z   test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8838329Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81185
2022-11-23T03:04:41.8838552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81186
2022-11-23T03:04:41.8838765Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81187
2022-11-23T03:04:41.8838960Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81188
2022-11-23T03:04:41.8839335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8839508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8839885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8840074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8840436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8840613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8840983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8841155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8841559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8841739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8842113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8842298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8842661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8842833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8843205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8843390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8843630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwjjjaqn2
2022-11-23T03:04:41.8843906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwjjjaqn2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8844158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpah3n0xem
2022-11-23T03:04:41.8844425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpah3n0xem/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8844677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22o1crai
2022-11-23T03:04:41.8844942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22o1crai/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8845244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkybrpmr2
2022-11-23T03:04:41.8845510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkybrpmr2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8845719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8845956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8846180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8846404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8846504Z ok (4.441s)
2022-11-23T03:04:41.8846524Z 
2022-11-23T03:04:41.8846791Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8846903Z Ran 1 test in 4.442s
2022-11-23T03:04:41.8846923Z 
2022-11-23T03:04:41.8847015Z OK
2022-11-23T03:04:41.8847037Z 
2022-11-23T03:04:41.8847159Z Generating XML reports...
2022-11-23T03:04:41.8847691Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024835.xml
2022-11-23T03:04:41.8848057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8848233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8848612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8848801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8849053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7itzql7
2022-11-23T03:04:41.8849321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7itzql7/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8849344Z 
2022-11-23T03:04:41.8849453Z Running tests...
2022-11-23T03:04:41.8849699Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8850054Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8850468Z   test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8850692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81356
2022-11-23T03:04:41.8850906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81357
2022-11-23T03:04:41.8851120Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81358
2022-11-23T03:04:41.8851332Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81359
2022-11-23T03:04:41.8851708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8851888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8852247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8852436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8852803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8852974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8853342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8853530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8853886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8854055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8854462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8854649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8855015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8855184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8855550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8855736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8855990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5lqo86dt
2022-11-23T03:04:41.8856258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5lqo86dt/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8856516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ct083dp
2022-11-23T03:04:41.8856753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpivvcx9tf
2022-11-23T03:04:41.8857018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ct083dp/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8857283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpivvcx9tf/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8857533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqwre4sf5
2022-11-23T03:04:41.8857797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqwre4sf5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8858022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8858250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8858478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8858686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8858786Z ok (4.432s)
2022-11-23T03:04:41.8858806Z 
2022-11-23T03:04:41.8859076Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8859234Z Ran 1 test in 4.432s
2022-11-23T03:04:41.8859255Z 
2022-11-23T03:04:41.8859352Z OK
2022-11-23T03:04:41.8859370Z 
2022-11-23T03:04:41.8859492Z Generating XML reports...
2022-11-23T03:04:41.8860044Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024842.xml
2022-11-23T03:04:41.8860416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8860590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8860955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8861144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8861396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdvss62i2
2022-11-23T03:04:41.8861666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdvss62i2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8861686Z 
2022-11-23T03:04:41.8861794Z Running tests...
2022-11-23T03:04:41.8862055Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8862414Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8862735Z   test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8862953Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81527
2022-11-23T03:04:41.8863208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81528
2022-11-23T03:04:41.8863421Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81529
2022-11-23T03:04:41.8863629Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81530
2022-11-23T03:04:41.8864010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8864186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8864564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8864753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8865116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8865274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8865647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8865834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8866196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8866369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8866743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8867095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8867469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8867640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8868004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8868190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8868445Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_pf1uvj
2022-11-23T03:04:41.8868806Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_pf1uvj/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8869069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_yktakv
2022-11-23T03:04:41.8869334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_yktakv/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8869585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmv5fixnt
2022-11-23T03:04:41.8869851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmv5fixnt/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8870087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjd9f03xm
2022-11-23T03:04:41.8870353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjd9f03xm/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8870581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8870810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8871035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8871257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8871361Z ok (6.128s)
2022-11-23T03:04:41.8871381Z 
2022-11-23T03:04:41.8871653Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8871765Z Ran 1 test in 6.128s
2022-11-23T03:04:41.8871784Z 
2022-11-23T03:04:41.8871859Z OK
2022-11-23T03:04:41.8871942Z 
2022-11-23T03:04:41.8872072Z Generating XML reports...
2022-11-23T03:04:41.8872626Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024849.xml
2022-11-23T03:04:41.8872997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8873174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8873553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8873747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8874000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprx80cifu
2022-11-23T03:04:41.8874252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprx80cifu/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8874292Z 
2022-11-23T03:04:41.8874383Z Running tests...
2022-11-23T03:04:41.8874646Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8875004Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8875317Z   test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8875533Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81702
2022-11-23T03:04:41.8875750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81703
2022-11-23T03:04:41.8875964Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81704
2022-11-23T03:04:41.8876173Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81705
2022-11-23T03:04:41.8876530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8876707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8877083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8877274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8877680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8877858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8878230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8878418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8878757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8878926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8879298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8879483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8879852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8880023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8880389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8880572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8880827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_f57sepo
2022-11-23T03:04:41.8881081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_f57sepo/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8881386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi47nq2v2
2022-11-23T03:04:41.8881649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi47nq2v2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8881900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpovnpivwu
2022-11-23T03:04:41.8882170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpovnpivwu/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8882418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9utk9jd
2022-11-23T03:04:41.8882691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9utk9jd/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8882903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8883139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8883368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8883590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8883689Z ok (6.005s)
2022-11-23T03:04:41.8883710Z 
2022-11-23T03:04:41.8883977Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8884090Z Ran 1 test in 6.005s
2022-11-23T03:04:41.8884110Z 
2022-11-23T03:04:41.8884201Z OK
2022-11-23T03:04:41.8884220Z 
2022-11-23T03:04:41.8884324Z Generating XML reports...
2022-11-23T03:04:41.8884873Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024858.xml
2022-11-23T03:04:41.8885240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8885412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8885795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8885983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8886234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeb5uvbw4
2022-11-23T03:04:41.8886553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeb5uvbw4/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8886574Z 
2022-11-23T03:04:41.8886684Z Running tests...
2022-11-23T03:04:41.8886930Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8887288Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8887600Z   test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8887816Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81877
2022-11-23T03:04:41.8888037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81878
2022-11-23T03:04:41.8888251Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81879
2022-11-23T03:04:41.8888463Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81880
2022-11-23T03:04:41.8888833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8888993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8889373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8889561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8889919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8890144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8890520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8890709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8891070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8891242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8891594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8891780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8892145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8892314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8892684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8892868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8893122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3sqgw0q5
2022-11-23T03:04:41.8893393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3sqgw0q5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8893626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_zzthkm
2022-11-23T03:04:41.8893891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_zzthkm/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8894139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcujkt6jr
2022-11-23T03:04:41.8894400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcujkt6jr/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8894650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcb7mmyvu
2022-11-23T03:04:41.8894912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcb7mmyvu/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8895141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8895420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8895648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8895850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8895951Z ok (11.342s)
2022-11-23T03:04:41.8895970Z 
2022-11-23T03:04:41.8896238Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8896351Z Ran 1 test in 11.342s
2022-11-23T03:04:41.8896371Z 
2022-11-23T03:04:41.8896468Z OK
2022-11-23T03:04:41.8896487Z 
2022-11-23T03:04:41.8896611Z Generating XML reports...
2022-11-23T03:04:41.8897156Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024906.xml
2022-11-23T03:04:41.8897524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8897683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8898059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8898247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8898499Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprk9kgjp_
2022-11-23T03:04:41.8898763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprk9kgjp_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8898832Z 
2022-11-23T03:04:41.8898943Z Running tests...
2022-11-23T03:04:41.8899207Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8899567Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8899861Z   test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8900610Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79750 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2022-11-23T03:04:41.8900645Z 
2022-11-23T03:04:41.8900888Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8901000Z Ran 1 test in 1.627s
2022-11-23T03:04:41.8901019Z 
2022-11-23T03:04:41.8901125Z OK (skipped=1)
2022-11-23T03:04:41.8901148Z 
2022-11-23T03:04:41.8901270Z Generating XML reports...
2022-11-23T03:04:41.8901863Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024920.xml
2022-11-23T03:04:41.8902238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8902413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8902791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8902980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8903218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81swuphm
2022-11-23T03:04:41.8903481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81swuphm/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8903505Z 
2022-11-23T03:04:41.8903610Z Running tests...
2022-11-23T03:04:41.8903872Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8904226Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8904575Z   test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8904799Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82090
2022-11-23T03:04:41.8905016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82091
2022-11-23T03:04:41.8905213Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82092
2022-11-23T03:04:41.8905422Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82093
2022-11-23T03:04:41.8905799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8905976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8906352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8906539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8907069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8907251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8907629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8907799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8908162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8908330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8908790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8908974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8909334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8909508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8909874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8910040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8910296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpic9gri_6
2022-11-23T03:04:41.8910564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpic9gri_6/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8910854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gn9f0o5
2022-11-23T03:04:41.8911121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gn9f0o5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8911368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurqcwiun
2022-11-23T03:04:41.8911638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurqcwiun/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8911886Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2uecgwmt
2022-11-23T03:04:41.8912148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2uecgwmt/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8912359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8912586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8912814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8913038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8913185Z fi_getinfo: -61
2022-11-23T03:04:41.8913319Z fi_getinfo: -61
2022-11-23T03:04:41.8913452Z fi_getinfo: -61
2022-11-23T03:04:41.8913569Z fi_getinfo: -61
2022-11-23T03:04:41.8913732Z ok (18.958s)
2022-11-23T03:04:41.8913753Z 
2022-11-23T03:04:41.8914027Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8914140Z Ran 1 test in 18.958s
2022-11-23T03:04:41.8914158Z 
2022-11-23T03:04:41.8914249Z OK
2022-11-23T03:04:41.8914268Z 
2022-11-23T03:04:41.8914390Z Generating XML reports...
2022-11-23T03:04:41.8914932Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024925.xml
2022-11-23T03:04:41.8915297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8915473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8915833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8916024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8916274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6a1f0ft
2022-11-23T03:04:41.8916535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6a1f0ft/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8916554Z 
2022-11-23T03:04:41.8916659Z Running tests...
2022-11-23T03:04:41.8916921Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8917276Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8917573Z   test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8917833Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82449
2022-11-23T03:04:41.8918050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82450
2022-11-23T03:04:41.8918268Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82451
2022-11-23T03:04:41.8918479Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82452
2022-11-23T03:04:41.8918850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8919023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8919400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8919591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8919957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8920111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8920480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8920671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8921026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8921195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8921563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8921748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8922111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8922268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8922636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8922877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8923140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmwvkofy5
2022-11-23T03:04:41.8923411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmwvkofy5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8923664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23bl7uao
2022-11-23T03:04:41.8923929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23bl7uao/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8924179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpetkkm4lt
2022-11-23T03:04:41.8924446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpetkkm4lt/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8924676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmvgmtch
2022-11-23T03:04:41.8924944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmvgmtch/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8925170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8925396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8925619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8925842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8925988Z fi_getinfo: -61
2022-11-23T03:04:41.8926121Z fi_getinfo: -61
2022-11-23T03:04:41.8926295Z fi_getinfo: -61
2022-11-23T03:04:41.8926429Z fi_getinfo: -61
2022-11-23T03:04:41.8926528Z ok (13.255s)
2022-11-23T03:04:41.8926548Z 
2022-11-23T03:04:41.8926808Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8926919Z Ran 1 test in 13.255s
2022-11-23T03:04:41.8926937Z 
2022-11-23T03:04:41.8927028Z OK
2022-11-23T03:04:41.8927047Z 
2022-11-23T03:04:41.8927172Z Generating XML reports...
2022-11-23T03:04:41.8927719Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024946.xml
2022-11-23T03:04:41.8928070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8928243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8928617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8928809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8929059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3p6g07b
2022-11-23T03:04:41.8929324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3p6g07b/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8929343Z 
2022-11-23T03:04:41.8929452Z Running tests...
2022-11-23T03:04:41.8929712Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8930051Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8930359Z   test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8930578Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82808
2022-11-23T03:04:41.8930794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82809
2022-11-23T03:04:41.8931009Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82810
2022-11-23T03:04:41.8931218Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82811
2022-11-23T03:04:41.8931589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8931825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8932220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8932391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8932753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8932924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8933294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8933485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8933839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8934008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8934379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8934551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8934905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8935073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8935439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8935679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8935934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7e_tdol8
2022-11-23T03:04:41.8936201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7e_tdol8/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8936456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazedfmzn
2022-11-23T03:04:41.8936723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazedfmzn/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8936955Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2diivyj
2022-11-23T03:04:41.8937217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2diivyj/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8937465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpez3s5f63
2022-11-23T03:04:41.8937734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpez3s5f63/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8937962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8938185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8938413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8938632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8938761Z fi_getinfo: -61
2022-11-23T03:04:41.8938895Z fi_getinfo: -61
2022-11-23T03:04:41.8939029Z fi_getinfo: -61
2022-11-23T03:04:41.8939161Z fi_getinfo: -61
2022-11-23T03:04:41.8939259Z ok (11.631s)
2022-11-23T03:04:41.8939278Z 
2022-11-23T03:04:41.8939540Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8939652Z Ran 1 test in 11.632s
2022-11-23T03:04:41.8939670Z 
2022-11-23T03:04:41.8939764Z OK
2022-11-23T03:04:41.8939783Z 
2022-11-23T03:04:41.8939887Z Generating XML reports...
2022-11-23T03:04:41.8940434Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025002.xml
2022-11-23T03:04:41.8940854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8941050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8941419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8941611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8941862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpukg64x6d
2022-11-23T03:04:41.8944150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpukg64x6d/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8944177Z 
2022-11-23T03:04:41.8944286Z Running tests...
2022-11-23T03:04:41.8944541Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8944900Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8945198Z   test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8945419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83162
2022-11-23T03:04:41.8945636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83163
2022-11-23T03:04:41.8945850Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83164
2022-11-23T03:04:41.8946063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83165
2022-11-23T03:04:41.8946442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8946663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8947570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8947773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8948152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8948325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8948696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8948884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8949247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8949424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8949774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8949960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8950323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8950497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8950863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8951048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8951306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqyetct_q
2022-11-23T03:04:41.8951578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqyetct_q/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8951838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujlwu34_
2022-11-23T03:04:41.8952096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujlwu34_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8952339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp95e0u2go
2022-11-23T03:04:41.8952705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp95e0u2go/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8952969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8i7q6uu
2022-11-23T03:04:41.8953235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8i7q6uu/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8953467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8953691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8953922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8954132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8954282Z fi_getinfo: -61
2022-11-23T03:04:41.8954418Z fi_getinfo: -61
2022-11-23T03:04:41.8954551Z fi_getinfo: -61
2022-11-23T03:04:41.8954689Z fi_getinfo: -61
2022-11-23T03:04:41.8954791Z ok (4.838s)
2022-11-23T03:04:41.8954811Z 
2022-11-23T03:04:41.8955076Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8955170Z Ran 1 test in 4.838s
2022-11-23T03:04:41.8955189Z 
2022-11-23T03:04:41.8955281Z OK
2022-11-23T03:04:41.8955300Z 
2022-11-23T03:04:41.8955421Z Generating XML reports...
2022-11-23T03:04:41.8955965Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025017.xml
2022-11-23T03:04:41.8956333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8956587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8956972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8957167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8957424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppywujj2i
2022-11-23T03:04:41.8957676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppywujj2i/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8957696Z 
2022-11-23T03:04:41.8957804Z Running tests...
2022-11-23T03:04:41.8958064Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8958422Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8958742Z   test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8958959Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83505
2022-11-23T03:04:41.8959177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83506
2022-11-23T03:04:41.8959395Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83507
2022-11-23T03:04:41.8959589Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83508
2022-11-23T03:04:41.8959959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8960134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8960513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8960705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8961071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8961246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8961667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8961862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8962210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8962383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8962749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8962936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8963305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8963476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8963843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8964034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8964272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qyn7xp0
2022-11-23T03:04:41.8964541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qyn7xp0/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8964796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyhdhzq7
2022-11-23T03:04:41.8965063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyhdhzq7/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8965313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8f_pdvsk
2022-11-23T03:04:41.8965635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8f_pdvsk/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8965888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphp5mk7rz
2022-11-23T03:04:41.8966157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphp5mk7rz/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8966389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8966599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8966827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8967051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8967196Z fi_getinfo: -61
2022-11-23T03:04:41.8967333Z fi_getinfo: -61
2022-11-23T03:04:41.8967472Z fi_getinfo: -61
2022-11-23T03:04:41.8967605Z fi_getinfo: -61
2022-11-23T03:04:41.8967686Z ok (7.959s)
2022-11-23T03:04:41.8967706Z 
2022-11-23T03:04:41.8967968Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8968079Z Ran 1 test in 7.959s
2022-11-23T03:04:41.8968098Z 
2022-11-23T03:04:41.8968190Z OK
2022-11-23T03:04:41.8968208Z 
2022-11-23T03:04:41.8968332Z Generating XML reports...
2022-11-23T03:04:41.8968876Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025024.xml
2022-11-23T03:04:41.8969248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8969422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8969800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8969978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8970234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppp0j6jpf
2022-11-23T03:04:41.8970503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppp0j6jpf/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8970522Z 
2022-11-23T03:04:41.8970678Z Running tests...
2022-11-23T03:04:41.8970949Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8971306Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8971625Z   test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8971843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83856
2022-11-23T03:04:41.8972042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83857
2022-11-23T03:04:41.8972262Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83858
2022-11-23T03:04:41.8972471Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83859
2022-11-23T03:04:41.8972846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8973020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8973400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8973586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8973942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8974113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8974468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8974714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8975074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8975248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8975617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8975800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8976163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8976333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8976683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8976874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8977130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_gaoaqwi
2022-11-23T03:04:41.8977399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_gaoaqwi/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8977655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsowdq0m2
2022-11-23T03:04:41.8977923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsowdq0m2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8978173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_8pfm61
2022-11-23T03:04:41.8978433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_8pfm61/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8978683Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5ap95_t
2022-11-23T03:04:41.8978933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5ap95_t/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8979163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8979393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8979667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8979899Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8980042Z fi_getinfo: -61
2022-11-23T03:04:41.8980179Z fi_getinfo: -61
2022-11-23T03:04:41.8980297Z fi_getinfo: -61
2022-11-23T03:04:41.8980430Z fi_getinfo: -61
2022-11-23T03:04:41.8980528Z ok (7.834s)
2022-11-23T03:04:41.8980548Z 
2022-11-23T03:04:41.8980809Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8980919Z Ran 1 test in 7.834s
2022-11-23T03:04:41.8980943Z 
2022-11-23T03:04:41.8981035Z OK
2022-11-23T03:04:41.8981054Z 
2022-11-23T03:04:41.8981176Z Generating XML reports...
2022-11-23T03:04:41.8981727Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025035.xml
2022-11-23T03:04:41.8982099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8982258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8982637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8982825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8983081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3p5rwdhf
2022-11-23T03:04:41.8983349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3p5rwdhf/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8983417Z 
2022-11-23T03:04:41.8983529Z Running tests...
2022-11-23T03:04:41.8983795Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8984153Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8984446Z   test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8984666Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84207
2022-11-23T03:04:41.8984878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84208
2022-11-23T03:04:41.8985092Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84209
2022-11-23T03:04:41.8985303Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84210
2022-11-23T03:04:41.8985676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8985855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8986233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8986424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8986949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8987128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8987511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8987700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8988058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8988232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8988596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8988781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8989206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8989389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8989760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8989942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8990198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucywfzd4
2022-11-23T03:04:41.8990466Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucywfzd4/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8990725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_1o8_da
2022-11-23T03:04:41.8990991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_1o8_da/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8991248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptk6tqmm
2022-11-23T03:04:41.8991497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptk6tqmm/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8991749Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcc8_csre
2022-11-23T03:04:41.8992009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcc8_csre/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8992238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.8992466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.8992821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.8993048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.8993195Z fi_getinfo: -61
2022-11-23T03:04:41.8993314Z fi_getinfo: -61
2022-11-23T03:04:41.8993450Z fi_getinfo: -61
2022-11-23T03:04:41.8993588Z fi_getinfo: -61
2022-11-23T03:04:41.8993688Z ok (7.843s)
2022-11-23T03:04:41.8993707Z 
2022-11-23T03:04:41.8993969Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8994080Z Ran 1 test in 7.844s
2022-11-23T03:04:41.8994099Z 
2022-11-23T03:04:41.8994187Z OK
2022-11-23T03:04:41.8994206Z 
2022-11-23T03:04:41.8994310Z Generating XML reports...
2022-11-23T03:04:41.8994858Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025045.xml
2022-11-23T03:04:41.8995232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.8995407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.8995783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.8995977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.8996231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7metgw6
2022-11-23T03:04:41.8996502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7metgw6/_remote_module_non_scriptable.py
2022-11-23T03:04:41.8996521Z 
2022-11-23T03:04:41.8996629Z Running tests...
2022-11-23T03:04:41.8996875Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8997235Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.8997565Z   test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.8998387Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/80008 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2022-11-23T03:04:41.8998411Z 
2022-11-23T03:04:41.8998680Z ----------------------------------------------------------------------
2022-11-23T03:04:41.8998790Z Ran 1 test in 1.627s
2022-11-23T03:04:41.8998810Z 
2022-11-23T03:04:41.8998915Z OK (skipped=1)
2022-11-23T03:04:41.8998934Z 
2022-11-23T03:04:41.8999055Z Generating XML reports...
2022-11-23T03:04:41.8999595Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025056.xml
2022-11-23T03:04:41.8999964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9000123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9000497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9000689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9000943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbrqom91h
2022-11-23T03:04:41.9001211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbrqom91h/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9001231Z 
2022-11-23T03:04:41.9001338Z Running tests...
2022-11-23T03:04:41.9001601Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9001956Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9002313Z   test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9002514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84588
2022-11-23T03:04:41.9002731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84589
2022-11-23T03:04:41.9002949Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84590
2022-11-23T03:04:41.9003157Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84591
2022-11-23T03:04:41.9003527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9003700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9004064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9004241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9004603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9004793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9005174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9005360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9005727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9005900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9006274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9006459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9006827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9006982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9007399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9007591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9007848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmq6m993
2022-11-23T03:04:41.9008117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmq6m993/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9008370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplb0y29ek
2022-11-23T03:04:41.9008635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplb0y29ek/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9008887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3wv06gm
2022-11-23T03:04:41.9009133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3wv06gm/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9009384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaxirq29o
2022-11-23T03:04:41.9009649Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaxirq29o/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9009878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9010105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9010326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9010553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9010702Z fi_getinfo: -61
2022-11-23T03:04:41.9010931Z fi_getinfo: -61
2022-11-23T03:04:41.9011069Z fi_getinfo: -61
2022-11-23T03:04:41.9011204Z fi_getinfo: -61
2022-11-23T03:04:41.9011301Z ok (10.347s)
2022-11-23T03:04:41.9011321Z 
2022-11-23T03:04:41.9011580Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9011689Z Ran 1 test in 10.347s
2022-11-23T03:04:41.9011708Z 
2022-11-23T03:04:41.9011804Z OK
2022-11-23T03:04:41.9011823Z 
2022-11-23T03:04:41.9011946Z Generating XML reports...
2022-11-23T03:04:41.9012478Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025100.xml
2022-11-23T03:04:41.9012847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9013024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9013401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9013597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9013853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnpiwx2dc
2022-11-23T03:04:41.9014128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnpiwx2dc/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9014147Z 
2022-11-23T03:04:41.9014255Z Running tests...
2022-11-23T03:04:41.9014500Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9014860Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9015165Z   test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9015386Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84939
2022-11-23T03:04:41.9015603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84940
2022-11-23T03:04:41.9015820Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84941
2022-11-23T03:04:41.9016032Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84942
2022-11-23T03:04:41.9016453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9016634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9016998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9017189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9017549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9017722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9018101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9018287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9018653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9018832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9019208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9019376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9019739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9019912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9020279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9020518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9020772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpio_n8h9_
2022-11-23T03:04:41.9021043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpio_n8h9_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9021299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmavui0o2
2022-11-23T03:04:41.9021551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmavui0o2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9021805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsowiqviu
2022-11-23T03:04:41.9022072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsowiqviu/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9022323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1i_e87u
2022-11-23T03:04:41.9022589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1i_e87u/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9022818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9023051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9023278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9023508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9023641Z fi_getinfo: -61
2022-11-23T03:04:41.9023776Z fi_getinfo: -61
2022-11-23T03:04:41.9023911Z fi_getinfo: -61
2022-11-23T03:04:41.9024047Z fi_getinfo: -61
2022-11-23T03:04:41.9024145Z ok (10.355s)
2022-11-23T03:04:41.9024165Z 
2022-11-23T03:04:41.9024426Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9024543Z Ran 1 test in 10.355s
2022-11-23T03:04:41.9024563Z 
2022-11-23T03:04:41.9024636Z OK
2022-11-23T03:04:41.9024655Z 
2022-11-23T03:04:41.9024778Z Generating XML reports...
2022-11-23T03:04:41.9025324Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025113.xml
2022-11-23T03:04:41.9025748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9025929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9026312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9026501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9026915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8c3_5o3
2022-11-23T03:04:41.9027188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8c3_5o3/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9027231Z 
2022-11-23T03:04:41.9027321Z Running tests...
2022-11-23T03:04:41.9027594Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9027952Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9028260Z   test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9028479Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85290
2022-11-23T03:04:41.9028693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85291
2022-11-23T03:04:41.9028907Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85292
2022-11-23T03:04:41.9029115Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85293
2022-11-23T03:04:41.9029471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9029736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9030119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9030312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9030673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9030846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9031211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9031384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9031743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9031939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9032313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9032502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9032863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9033038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9033413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9033599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9033854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hxrbhao
2022-11-23T03:04:41.9034110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hxrbhao/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9034369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj1ud0ot
2022-11-23T03:04:41.9034641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj1ud0ot/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9034959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnq7m_5hf
2022-11-23T03:04:41.9035234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnq7m_5hf/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9035485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58yzajan
2022-11-23T03:04:41.9035747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58yzajan/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9035975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9036201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9036412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9036639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9036785Z fi_getinfo: -61
2022-11-23T03:04:41.9036922Z fi_getinfo: -61
2022-11-23T03:04:41.9037060Z fi_getinfo: -61
2022-11-23T03:04:41.9037196Z fi_getinfo: -61
2022-11-23T03:04:41.9037294Z ok (10.238s)
2022-11-23T03:04:41.9037314Z 
2022-11-23T03:04:41.9037560Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9037672Z Ran 1 test in 10.239s
2022-11-23T03:04:41.9037692Z 
2022-11-23T03:04:41.9037783Z OK
2022-11-23T03:04:41.9037801Z 
2022-11-23T03:04:41.9037923Z Generating XML reports...
2022-11-23T03:04:41.9038469Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025126.xml
2022-11-23T03:04:41.9038896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9039070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9039450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9039623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9039879Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbaqmkxth
2022-11-23T03:04:41.9040146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbaqmkxth/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9040165Z 
2022-11-23T03:04:41.9040273Z Running tests...
2022-11-23T03:04:41.9040534Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9040891Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9041194Z   test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9041410Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85641
2022-11-23T03:04:41.9041630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85642
2022-11-23T03:04:41.9041827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85643
2022-11-23T03:04:41.9042038Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85644
2022-11-23T03:04:41.9042409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9042583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9042961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9043156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9043520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9043693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9044096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9044293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9044659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9044829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9045194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9045381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9045751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9045921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9046289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9046457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9046711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyo1qh7l
2022-11-23T03:04:41.9046983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyo1qh7l/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9047234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7oyd1je2
2022-11-23T03:04:41.9047500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7oyd1je2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9047813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpniysd2py
2022-11-23T03:04:41.9048330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpniysd2py/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9048650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw1ztq0kt
2022-11-23T03:04:41.9048925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw1ztq0kt/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9049136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9049363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9049588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9049813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9049976Z fi_getinfo: -61
2022-11-23T03:04:41.9050113Z fi_getinfo: -61
2022-11-23T03:04:41.9050247Z fi_getinfo: -61
2022-11-23T03:04:41.9050364Z fi_getinfo: -61
2022-11-23T03:04:41.9050464Z ok (10.425s)
2022-11-23T03:04:41.9050484Z 
2022-11-23T03:04:41.9050746Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9050856Z Ran 1 test in 10.425s
2022-11-23T03:04:41.9050880Z 
2022-11-23T03:04:41.9050974Z OK
2022-11-23T03:04:41.9050993Z 
2022-11-23T03:04:41.9051117Z Generating XML reports...
2022-11-23T03:04:41.9051667Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025139.xml
2022-11-23T03:04:41.9052038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9052194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9052574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9052769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9053023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm5vv0vdr
2022-11-23T03:04:41.9053352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm5vv0vdr/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9053374Z 
2022-11-23T03:04:41.9053485Z Running tests...
2022-11-23T03:04:41.9053753Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9054113Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9054416Z   test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9054617Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85992
2022-11-23T03:04:41.9054837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85993
2022-11-23T03:04:41.9055054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85994
2022-11-23T03:04:41.9055263Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85995
2022-11-23T03:04:41.9055640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9055816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9056198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9056388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9056730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9056900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9057329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9057519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9057874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9058049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9058419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9058607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9058971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9059123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9059489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9059679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9059934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpume4b5eu
2022-11-23T03:04:41.9060208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpume4b5eu/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9060460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu6kohni9
2022-11-23T03:04:41.9060712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpomz96ay2
2022-11-23T03:04:41.9060978Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu6kohni9/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9061220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpomz96ay2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9061475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_pceaph
2022-11-23T03:04:41.9061746Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_pceaph/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9061975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9062273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9062506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9062728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9062874Z fi_getinfo: -61
2022-11-23T03:04:41.9063009Z fi_getinfo: -61
2022-11-23T03:04:41.9063127Z fi_getinfo: -61
2022-11-23T03:04:41.9063261Z fi_getinfo: -61
2022-11-23T03:04:41.9063360Z ok (10.348s)
2022-11-23T03:04:41.9063379Z 
2022-11-23T03:04:41.9063644Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9063761Z Ran 1 test in 10.348s
2022-11-23T03:04:41.9063780Z 
2022-11-23T03:04:41.9063871Z OK
2022-11-23T03:04:41.9063890Z 
2022-11-23T03:04:41.9064009Z Generating XML reports...
2022-11-23T03:04:41.9064537Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025152.xml
2022-11-23T03:04:41.9064906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9065078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9065456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9065645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9065898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvqvmb_2r
2022-11-23T03:04:41.9066162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvqvmb_2r/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9066232Z 
2022-11-23T03:04:41.9066344Z Running tests...
2022-11-23T03:04:41.9066607Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9067125Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9067450Z   test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9067660Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86351
2022-11-23T03:04:41.9067874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86352
2022-11-23T03:04:41.9068087Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86353
2022-11-23T03:04:41.9068297Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86354
2022-11-23T03:04:41.9068679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9068853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9069213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9069408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9069774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9069944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9070319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9070506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9070869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9071044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9071417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9071667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9072047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9072218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9072584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9072771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9073029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn74btojx
2022-11-23T03:04:41.9073301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn74btojx/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9073552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fuqmnl9
2022-11-23T03:04:41.9073802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fuqmnl9/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9074058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfquyudvt
2022-11-23T03:04:41.9074323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfquyudvt/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9074572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdjhmvce8
2022-11-23T03:04:41.9074838Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdjhmvce8/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9075068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9075364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9075592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9075816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9075948Z fi_getinfo: -61
2022-11-23T03:04:41.9076088Z fi_getinfo: -61
2022-11-23T03:04:41.9076225Z fi_getinfo: -61
2022-11-23T03:04:41.9076358Z fi_getinfo: -61
2022-11-23T03:04:41.9076456Z ok (10.442s)
2022-11-23T03:04:41.9076475Z 
2022-11-23T03:04:41.9076739Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9076853Z Ran 1 test in 10.442s
2022-11-23T03:04:41.9076872Z 
2022-11-23T03:04:41.9076946Z OK
2022-11-23T03:04:41.9076965Z 
2022-11-23T03:04:41.9077087Z Generating XML reports...
2022-11-23T03:04:41.9077627Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025205.xml
2022-11-23T03:04:41.9078000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9078174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9078557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9078747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9079001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0c213by
2022-11-23T03:04:41.9079267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0c213by/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9079287Z 
2022-11-23T03:04:41.9079377Z Running tests...
2022-11-23T03:04:41.9079641Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9079995Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9080298Z   test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9080516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86710
2022-11-23T03:04:41.9080782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86711
2022-11-23T03:04:41.9081004Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86712
2022-11-23T03:04:41.9081215Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86713
2022-11-23T03:04:41.9081575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9081749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9082126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9082318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9082688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9082859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9083231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9083418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9083783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9083938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9084305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9084545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9084906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9085077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9085444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9085633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9085891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqn0fh7lv
2022-11-23T03:04:41.9086145Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqn0fh7lv/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9086396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sfe0qjh
2022-11-23T03:04:41.9086663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sfe0qjh/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9086918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2x_u38le
2022-11-23T03:04:41.9087169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnamlr4ge
2022-11-23T03:04:41.9087436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2x_u38le/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9087699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnamlr4ge/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9087927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9088143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9088352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9088578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9088731Z fi_getinfo: -61
2022-11-23T03:04:41.9088865Z fi_getinfo: -61
2022-11-23T03:04:41.9089001Z fi_getinfo: -61
2022-11-23T03:04:41.9089135Z fi_getinfo: -61
2022-11-23T03:04:41.9089236Z ok (10.246s)
2022-11-23T03:04:41.9089255Z 
2022-11-23T03:04:41.9089502Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9089617Z Ran 1 test in 10.247s
2022-11-23T03:04:41.9089685Z 
2022-11-23T03:04:41.9089781Z OK
2022-11-23T03:04:41.9089799Z 
2022-11-23T03:04:41.9089924Z Generating XML reports...
2022-11-23T03:04:41.9090477Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025218.xml
2022-11-23T03:04:41.9090847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9091024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9091403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9091599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9091838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpudr5lvix
2022-11-23T03:04:41.9092117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpudr5lvix/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9092137Z 
2022-11-23T03:04:41.9092246Z Running tests...
2022-11-23T03:04:41.9092516Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9092875Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9093176Z   test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9093394Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87069
2022-11-23T03:04:41.9093664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87070
2022-11-23T03:04:41.9093862Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87071
2022-11-23T03:04:41.9094074Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87072
2022-11-23T03:04:41.9094453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9094628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9095006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9095194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9095554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9095724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9096101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9096269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9096636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9096809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9097181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9097367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9097726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9097900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9098272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9098438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9098695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3rxlx6j
2022-11-23T03:04:41.9099016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3rxlx6j/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9099273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1dvo5wr
2022-11-23T03:04:41.9099542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1dvo5wr/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9099794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vc5kxw_
2022-11-23T03:04:41.9100059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vc5kxw_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9100310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_wduj3o
2022-11-23T03:04:41.9100578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_wduj3o/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9100787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9101015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9101240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9101465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9101615Z fi_getinfo: -61
2022-11-23T03:04:41.9101754Z fi_getinfo: -61
2022-11-23T03:04:41.9101890Z fi_getinfo: -61
2022-11-23T03:04:41.9102007Z fi_getinfo: -61
2022-11-23T03:04:41.9102108Z ok (10.345s)
2022-11-23T03:04:41.9102126Z 
2022-11-23T03:04:41.9102389Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9102560Z Ran 1 test in 10.345s
2022-11-23T03:04:41.9102579Z 
2022-11-23T03:04:41.9102669Z OK
2022-11-23T03:04:41.9102688Z 
2022-11-23T03:04:41.9102811Z Generating XML reports...
2022-11-23T03:04:41.9103363Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025231.xml
2022-11-23T03:04:41.9103734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9103908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9104268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9104459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9104710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_iaqx_vh
2022-11-23T03:04:41.9104982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_iaqx_vh/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9105001Z 
2022-11-23T03:04:41.9105108Z Running tests...
2022-11-23T03:04:41.9105369Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9105731Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9106044Z   test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9106245Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87428
2022-11-23T03:04:41.9106458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87429
2022-11-23T03:04:41.9106672Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87430
2022-11-23T03:04:41.9107043Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87431
2022-11-23T03:04:41.9107438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9107615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9107995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9108260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9108641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9108797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9109169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9109356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9109716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9109894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9110263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9110454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9110859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9111013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9111382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9111567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9111826Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj9z2mo83
2022-11-23T03:04:41.9112164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj9z2mo83/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9112416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ebuf3j8
2022-11-23T03:04:41.9112682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ebuf3j8/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9112935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7os6nu7g
2022-11-23T03:04:41.9113197Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7os6nu7g/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9113428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5690dqlg
2022-11-23T03:04:41.9113692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5690dqlg/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9113917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9114145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9114373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9114598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9114747Z fi_getinfo: -61
2022-11-23T03:04:41.9114886Z fi_getinfo: -61
2022-11-23T03:04:41.9115004Z fi_getinfo: -61
2022-11-23T03:04:41.9115141Z fi_getinfo: -61
2022-11-23T03:04:41.9115240Z ok (10.255s)
2022-11-23T03:04:41.9115260Z 
2022-11-23T03:04:41.9115521Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9115633Z Ran 1 test in 10.255s
2022-11-23T03:04:41.9115651Z 
2022-11-23T03:04:41.9115743Z OK
2022-11-23T03:04:41.9115762Z 
2022-11-23T03:04:41.9115884Z Generating XML reports...
2022-11-23T03:04:41.9116432Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025244.xml
2022-11-23T03:04:41.9116791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9116967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9117391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9117587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9117838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdkoohe4l
2022-11-23T03:04:41.9118107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdkoohe4l/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9118126Z 
2022-11-23T03:04:41.9118233Z Running tests...
2022-11-23T03:04:41.9118495Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9118838Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9119157Z   test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9119373Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87779
2022-11-23T03:04:41.9119594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87780
2022-11-23T03:04:41.9119809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87781
2022-11-23T03:04:41.9120018Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87782
2022-11-23T03:04:41.9120390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9120563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9120939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9121167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9121531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9121699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9122070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9122259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9122616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9122784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9123158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9123333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9123693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9123868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9124245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9124432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9124688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf31wstua
2022-11-23T03:04:41.9124959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf31wstua/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9125209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7b_b05m
2022-11-23T03:04:41.9125473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7b_b05m/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9125711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplhy9dmg8
2022-11-23T03:04:41.9125977Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplhy9dmg8/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9126288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3pg2jan
2022-11-23T03:04:41.9126561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3pg2jan/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9126790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9127018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9127245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9127468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9127604Z fi_getinfo: -61
2022-11-23T03:04:41.9127734Z fi_getinfo: -61
2022-11-23T03:04:41.9127867Z fi_getinfo: -61
2022-11-23T03:04:41.9128001Z fi_getinfo: -61
2022-11-23T03:04:41.9128099Z ok (10.260s)
2022-11-23T03:04:41.9128119Z 
2022-11-23T03:04:41.9128378Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9128495Z Ran 1 test in 10.260s
2022-11-23T03:04:41.9128514Z 
2022-11-23T03:04:41.9128588Z OK
2022-11-23T03:04:41.9128622Z 
2022-11-23T03:04:41.9128727Z Generating XML reports...
2022-11-23T03:04:41.9129272Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025257.xml
2022-11-23T03:04:41.9129639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9129815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9130253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9130445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9130700Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpju6cu0sn
2022-11-23T03:04:41.9130971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpju6cu0sn/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9130991Z 
2022-11-23T03:04:41.9131081Z Running tests...
2022-11-23T03:04:41.9131345Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9131700Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9132007Z   test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9132225Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88130
2022-11-23T03:04:41.9132443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88131
2022-11-23T03:04:41.9132657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88132
2022-11-23T03:04:41.9132869Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88133
2022-11-23T03:04:41.9133245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9133403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9133784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9133971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9134333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9134508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9134881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9135071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9135523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9135684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9136059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9136245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9136611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9136782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9137156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9137340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9137596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0zdej3p
2022-11-23T03:04:41.9137873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0zdej3p/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9138107Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5bzkvloq
2022-11-23T03:04:41.9138375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5bzkvloq/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9138627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplscqp97n
2022-11-23T03:04:41.9138896Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplscqp97n/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9139200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj46w0boq
2022-11-23T03:04:41.9139464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj46w0boq/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9139692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9139924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9140134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9140360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9140504Z fi_getinfo: -61
2022-11-23T03:04:41.9140639Z fi_getinfo: -61
2022-11-23T03:04:41.9140775Z fi_getinfo: -61
2022-11-23T03:04:41.9140909Z fi_getinfo: -61
2022-11-23T03:04:41.9141007Z ok (10.361s)
2022-11-23T03:04:41.9141027Z 
2022-11-23T03:04:41.9141289Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9141387Z Ran 1 test in 10.361s
2022-11-23T03:04:41.9141406Z 
2022-11-23T03:04:41.9141497Z OK
2022-11-23T03:04:41.9141517Z 
2022-11-23T03:04:41.9141639Z Generating XML reports...
2022-11-23T03:04:41.9142191Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025310.xml
2022-11-23T03:04:41.9142567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9142741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9143120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9143312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9143547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7h4l2tvg
2022-11-23T03:04:41.9143818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7h4l2tvg/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9143838Z 
2022-11-23T03:04:41.9143946Z Running tests...
2022-11-23T03:04:41.9144213Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9144615Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9144930Z   test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9145147Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88481
2022-11-23T03:04:41.9145363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88482
2022-11-23T03:04:41.9145578Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88483
2022-11-23T03:04:41.9145774Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88484
2022-11-23T03:04:41.9146159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9146332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9146713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9147075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9147453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9147624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9147994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9148166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9148625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9148795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9149163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9149354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9149722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9149893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9150259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9150444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9150681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_rydxz4k
2022-11-23T03:04:41.9150953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_rydxz4k/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9151207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6iojicnf
2022-11-23T03:04:41.9151481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6iojicnf/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9151735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9y9kcdma
2022-11-23T03:04:41.9152002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9y9kcdma/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9152253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsitxrmj3
2022-11-23T03:04:41.9152516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsitxrmj3/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9152728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9152961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9153181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9153405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9153613Z fi_getinfo: -61
2022-11-23T03:04:41.9153763Z fi_getinfo: -61
2022-11-23T03:04:41.9153896Z fi_getinfo: -61
2022-11-23T03:04:41.9154031Z fi_getinfo: -61
2022-11-23T03:04:41.9154112Z ok (10.356s)
2022-11-23T03:04:41.9154132Z 
2022-11-23T03:04:41.9154396Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9154506Z Ran 1 test in 10.356s
2022-11-23T03:04:41.9154525Z 
2022-11-23T03:04:41.9154617Z OK
2022-11-23T03:04:41.9154636Z 
2022-11-23T03:04:41.9154757Z Generating XML reports...
2022-11-23T03:04:41.9155308Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025323.xml
2022-11-23T03:04:41.9155680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9155855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9156220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9156413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9156666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt71sm08v
2022-11-23T03:04:41.9156933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt71sm08v/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9156952Z 
2022-11-23T03:04:41.9157060Z Running tests...
2022-11-23T03:04:41.9157321Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9157739Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9158049Z   test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9158265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88832
2022-11-23T03:04:41.9158468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88833
2022-11-23T03:04:41.9158685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88834
2022-11-23T03:04:41.9158895Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88835
2022-11-23T03:04:41.9159268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9159443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9159819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9160013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9160374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9160531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9160909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9161096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9161459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9161636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9162008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9162199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9162560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9162729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9163126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9163318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9163576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9yjfuqo
2022-11-23T03:04:41.9163848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9yjfuqo/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9164103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbas9k1uf
2022-11-23T03:04:41.9164373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbas9k1uf/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9164623Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wl_eq0a
2022-11-23T03:04:41.9164887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wl_eq0a/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9165121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9gj2t57
2022-11-23T03:04:41.9165388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9gj2t57/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9165619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9165846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9166073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9166297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9166494Z fi_getinfo: -61
2022-11-23T03:04:41.9166632Z fi_getinfo: -61
2022-11-23T03:04:41.9166750Z fi_getinfo: -61
2022-11-23T03:04:41.9166883Z fi_getinfo: -61
2022-11-23T03:04:41.9166982Z ok (10.341s)
2022-11-23T03:04:41.9167001Z 
2022-11-23T03:04:41.9167268Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9167381Z Ran 1 test in 10.341s
2022-11-23T03:04:41.9167400Z 
2022-11-23T03:04:41.9167489Z OK
2022-11-23T03:04:41.9167508Z 
2022-11-23T03:04:41.9167627Z Generating XML reports...
2022-11-23T03:04:41.9168174Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025336.xml
2022-11-23T03:04:41.9168527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9168702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9169086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9169275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9169527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3yo74tw
2022-11-23T03:04:41.9169799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3yo74tw/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9169818Z 
2022-11-23T03:04:41.9169926Z Running tests...
2022-11-23T03:04:41.9170191Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9170546Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9170838Z   test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9171057Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89183
2022-11-23T03:04:41.9171278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89184
2022-11-23T03:04:41.9171495Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89185
2022-11-23T03:04:41.9171707Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89186
2022-11-23T03:04:41.9172122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9172303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9172684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9172856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9173218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9173393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9173766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9173954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9174313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9174485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9174853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9175039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9175389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9175560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9175987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9176173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9176435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnz8ukxj3
2022-11-23T03:04:41.9176704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnz8ukxj3/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9176959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtzsppwd
2022-11-23T03:04:41.9177230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtzsppwd/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9177467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpokk6qizw
2022-11-23T03:04:41.9177736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpokk6qizw/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9177992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj1u2hw6c
2022-11-23T03:04:41.9178251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj1u2hw6c/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9178480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9178709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9178933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9179162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9179311Z fi_getinfo: -61
2022-11-23T03:04:41.9179430Z fi_getinfo: -61
2022-11-23T03:04:41.9179564Z fi_getinfo: -61
2022-11-23T03:04:41.9179697Z fi_getinfo: -61
2022-11-23T03:04:41.9179795Z ok (10.266s)
2022-11-23T03:04:41.9179815Z 
2022-11-23T03:04:41.9180082Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9180193Z Ran 1 test in 10.266s
2022-11-23T03:04:41.9180212Z 
2022-11-23T03:04:41.9180304Z OK
2022-11-23T03:04:41.9180323Z 
2022-11-23T03:04:41.9180428Z Generating XML reports...
2022-11-23T03:04:41.9181024Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025349.xml
2022-11-23T03:04:41.9181405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9181580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9181957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9182147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9182399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_96xfsij
2022-11-23T03:04:41.9182671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_96xfsij/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9182691Z 
2022-11-23T03:04:41.9182798Z Running tests...
2022-11-23T03:04:41.9183044Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9183405Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9183715Z   test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9183932Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89534
2022-11-23T03:04:41.9184148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89535
2022-11-23T03:04:41.9184362Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89536
2022-11-23T03:04:41.9184578Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89537
2022-11-23T03:04:41.9185012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9185169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9185548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9185740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9186104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9186276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9186646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9187067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9187457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9187630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9187984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9188172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9188537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9188708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9189071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9189256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9189516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3nhl0ccs
2022-11-23T03:04:41.9189788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3nhl0ccs/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9190028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1mmghyyz
2022-11-23T03:04:41.9190398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1mmghyyz/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9190662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8jfs_h2e
2022-11-23T03:04:41.9190929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8jfs_h2e/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9191179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxrvbp70m
2022-11-23T03:04:41.9191443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxrvbp70m/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9191672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9191904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9192132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9192343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9192493Z fi_getinfo: -61
2022-11-23T03:04:41.9192629Z fi_getinfo: -61
2022-11-23T03:04:41.9192764Z fi_getinfo: -61
2022-11-23T03:04:41.9192898Z fi_getinfo: -61
2022-11-23T03:04:41.9192996Z ok (10.433s)
2022-11-23T03:04:41.9193016Z 
2022-11-23T03:04:41.9193283Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9193377Z Ran 1 test in 10.434s
2022-11-23T03:04:41.9193396Z 
2022-11-23T03:04:41.9193488Z OK
2022-11-23T03:04:41.9193506Z 
2022-11-23T03:04:41.9193626Z Generating XML reports...
2022-11-23T03:04:41.9194170Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025402.xml
2022-11-23T03:04:41.9194613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9194789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9195168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9195359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9195615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtrvco8l
2022-11-23T03:04:41.9195867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtrvco8l/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9195887Z 
2022-11-23T03:04:41.9195993Z Running tests...
2022-11-23T03:04:41.9196255Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9196622Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9196934Z   test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9197155Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89885
2022-11-23T03:04:41.9197372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89886
2022-11-23T03:04:41.9197587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89887
2022-11-23T03:04:41.9197779Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89888
2022-11-23T03:04:41.9198152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9198325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9198708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9198899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9199265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9199488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9199874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9200065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9200413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9200584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9200953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9201147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9201506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9201680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9202108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9202294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9202533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyh0967p1
2022-11-23T03:04:41.9202804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyh0967p1/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9203056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikwlnjnq
2022-11-23T03:04:41.9203362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwm4r6l4c
2022-11-23T03:04:41.9203631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikwlnjnq/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9203892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwm4r6l4c/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9204145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2du6rkvy
2022-11-23T03:04:41.9204409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2du6rkvy/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9204638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9204841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9205069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9205296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9205445Z fi_getinfo: -61
2022-11-23T03:04:41.9205582Z fi_getinfo: -61
2022-11-23T03:04:41.9205720Z fi_getinfo: -61
2022-11-23T03:04:41.9205856Z fi_getinfo: -61
2022-11-23T03:04:41.9205937Z ok (10.216s)
2022-11-23T03:04:41.9205956Z 
2022-11-23T03:04:41.9206223Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9206333Z Ran 1 test in 10.216s
2022-11-23T03:04:41.9206353Z 
2022-11-23T03:04:41.9206444Z OK
2022-11-23T03:04:41.9206463Z 
2022-11-23T03:04:41.9206584Z Generating XML reports...
2022-11-23T03:04:41.9207131Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025416.xml
2022-11-23T03:04:41.9207503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9207682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9208069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9208243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9208549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvi508q3
2022-11-23T03:04:41.9208822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvi508q3/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9208842Z 
2022-11-23T03:04:41.9208948Z Running tests...
2022-11-23T03:04:41.9209215Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9209571Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9209881Z   test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9210102Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90236
2022-11-23T03:04:41.9210300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90237
2022-11-23T03:04:41.9210515Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90238
2022-11-23T03:04:41.9210731Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90239
2022-11-23T03:04:41.9211140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9211315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9211691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9211880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9212240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9212469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9212827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9213015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9213376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9213547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9213920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9214106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9214463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9214638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9214988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9215178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9215440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrezxfre
2022-11-23T03:04:41.9215713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrezxfre/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9215968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkhtmnzp
2022-11-23T03:04:41.9216241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkhtmnzp/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9216490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuf9qm0r5
2022-11-23T03:04:41.9216754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuf9qm0r5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9217007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoakk66rx
2022-11-23T03:04:41.9217254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoakk66rx/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9217529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9217764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9217985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9218209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9218355Z fi_getinfo: -61
2022-11-23T03:04:41.9218491Z fi_getinfo: -61
2022-11-23T03:04:41.9218626Z fi_getinfo: -61
2022-11-23T03:04:41.9218745Z fi_getinfo: -61
2022-11-23T03:04:41.9218843Z ok (7.822s)
2022-11-23T03:04:41.9218866Z 
2022-11-23T03:04:41.9219127Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9219239Z Ran 1 test in 7.822s
2022-11-23T03:04:41.9219258Z 
2022-11-23T03:04:41.9219350Z OK
2022-11-23T03:04:41.9219369Z 
2022-11-23T03:04:41.9219491Z Generating XML reports...
2022-11-23T03:04:41.9220043Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025428.xml
2022-11-23T03:04:41.9220413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9220572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9220950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9221138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9221447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63v_l3l1
2022-11-23T03:04:41.9221718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63v_l3l1/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9221737Z 
2022-11-23T03:04:41.9221845Z Running tests...
2022-11-23T03:04:41.9222112Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9222475Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9222783Z   test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9223002Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90583
2022-11-23T03:04:41.9223218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90584
2022-11-23T03:04:41.9223432Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90585
2022-11-23T03:04:41.9223646Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90586
2022-11-23T03:04:41.9224017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9224192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9224572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9224761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9225110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9225280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9225647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9225838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9226200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9226373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9226980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9227191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9227547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9227715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9228087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9228273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9228534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo4cq54l9
2022-11-23T03:04:41.9228805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo4cq54l9/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9229055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52iqkbne
2022-11-23T03:04:41.9229325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52iqkbne/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9229581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntgpeyw9
2022-11-23T03:04:41.9229832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntgpeyw9/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9230084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpailts8_y
2022-11-23T03:04:41.9230352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpailts8_y/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9230661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9230884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9231108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9231338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9231486Z fi_getinfo: -61
2022-11-23T03:04:41.9231605Z fi_getinfo: -61
2022-11-23T03:04:41.9231740Z fi_getinfo: -61
2022-11-23T03:04:41.9231874Z fi_getinfo: -61
2022-11-23T03:04:41.9231974Z ok (10.364s)
2022-11-23T03:04:41.9231993Z 
2022-11-23T03:04:41.9232257Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9232369Z Ran 1 test in 10.365s
2022-11-23T03:04:41.9232388Z 
2022-11-23T03:04:41.9232478Z OK
2022-11-23T03:04:41.9232497Z 
2022-11-23T03:04:41.9232602Z Generating XML reports...
2022-11-23T03:04:41.9233157Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025439.xml
2022-11-23T03:04:41.9233526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9233705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9234083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9234274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9234530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfsaoluy
2022-11-23T03:04:41.9234801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfsaoluy/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9234821Z 
2022-11-23T03:04:41.9234929Z Running tests...
2022-11-23T03:04:41.9235175Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9235530Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9235846Z   test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9236127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90942
2022-11-23T03:04:41.9236351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90943
2022-11-23T03:04:41.9236565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90944
2022-11-23T03:04:41.9236779Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90945
2022-11-23T03:04:41.9237155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9237330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9237696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9237888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9238258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9238428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9238799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9238984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9239341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9239514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9239927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9240114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9240469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9240645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9241019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9241204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9241458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwg34c_tl
2022-11-23T03:04:41.9241712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemirotpy
2022-11-23T03:04:41.9241979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwg34c_tl/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9242231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemirotpy/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9242486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fj9v0xc
2022-11-23T03:04:41.9242754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fj9v0xc/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9243006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdk__x6jw
2022-11-23T03:04:41.9243270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdk__x6jw/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9243501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9243728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9243949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9244161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9244308Z fi_getinfo: -61
2022-11-23T03:04:41.9244443Z fi_getinfo: -61
2022-11-23T03:04:41.9244578Z fi_getinfo: -61
2022-11-23T03:04:41.9244711Z fi_getinfo: -61
2022-11-23T03:04:41.9244810Z ok (7.812s)
2022-11-23T03:04:41.9244830Z 
2022-11-23T03:04:41.9245144Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9245245Z Ran 1 test in 7.812s
2022-11-23T03:04:41.9245281Z 
2022-11-23T03:04:41.9245354Z OK
2022-11-23T03:04:41.9245373Z 
2022-11-23T03:04:41.9245496Z Generating XML reports...
2022-11-23T03:04:41.9246050Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025452.xml
2022-11-23T03:04:41.9246419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9246599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9246977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9247168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9247424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpln5mivn9
2022-11-23T03:04:41.9247672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpln5mivn9/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9247709Z 
2022-11-23T03:04:41.9247799Z Running tests...
2022-11-23T03:04:41.9248060Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9248418Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9248738Z   test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9249013Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91293
2022-11-23T03:04:41.9249230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91294
2022-11-23T03:04:41.9249445Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91295
2022-11-23T03:04:41.9249663Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91296
2022-11-23T03:04:41.9250019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9250193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9250569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9250758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9251124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9251302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9251673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9251863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9252208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9252380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9252747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9252931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9253288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9253464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9253837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9254023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9254338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ja03k2t
2022-11-23T03:04:41.9254585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdf3htyt
2022-11-23T03:04:41.9254854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ja03k2t/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9255117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdf3htyt/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9255366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1a0mozfe
2022-11-23T03:04:41.9255635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1a0mozfe/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9255885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqi5vtm2i
2022-11-23T03:04:41.9256149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqi5vtm2i/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9256380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9256585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9256811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9257034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9257181Z fi_getinfo: -61
2022-11-23T03:04:41.9257314Z fi_getinfo: -61
2022-11-23T03:04:41.9257448Z fi_getinfo: -61
2022-11-23T03:04:41.9257582Z fi_getinfo: -61
2022-11-23T03:04:41.9257718Z ok (7.814s)
2022-11-23T03:04:41.9257755Z 
2022-11-23T03:04:41.9258005Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9258115Z Ran 1 test in 7.814s
2022-11-23T03:04:41.9258134Z 
2022-11-23T03:04:41.9258226Z OK
2022-11-23T03:04:41.9258245Z 
2022-11-23T03:04:41.9258368Z Generating XML reports...
2022-11-23T03:04:41.9258918Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025503.xml
2022-11-23T03:04:41.9259288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9259464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9259842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9260015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9260279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjtsni050
2022-11-23T03:04:41.9260548Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjtsni050/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9260568Z 
2022-11-23T03:04:41.9260672Z Running tests...
2022-11-23T03:04:41.9260936Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9261294Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9261589Z   test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9261806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91644
2022-11-23T03:04:41.9262023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91645
2022-11-23T03:04:41.9262221Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91646
2022-11-23T03:04:41.9262440Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91647
2022-11-23T03:04:41.9262809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9262982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9263408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9263604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9263973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9264146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9264498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9264686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9265046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9265219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9265585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9265769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9266125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9266297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9266672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9267169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9267531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp267ws31k
2022-11-23T03:04:41.9267798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp267ws31k/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9268052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp425by594
2022-11-23T03:04:41.9268315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp425by594/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9268570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwnrln8t_
2022-11-23T03:04:41.9268837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwnrln8t_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9269089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxgsv7_k
2022-11-23T03:04:41.9269339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxgsv7_k/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9269575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9269799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9270025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9270255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9270414Z fi_getinfo: -61
2022-11-23T03:04:41.9270550Z fi_getinfo: -61
2022-11-23T03:04:41.9270686Z fi_getinfo: -61
2022-11-23T03:04:41.9270803Z fi_getinfo: -61
2022-11-23T03:04:41.9270903Z ok (10.466s)
2022-11-23T03:04:41.9270922Z 
2022-11-23T03:04:41.9271182Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9271294Z Ran 1 test in 10.466s
2022-11-23T03:04:41.9271313Z 
2022-11-23T03:04:41.9271405Z OK
2022-11-23T03:04:41.9271424Z 
2022-11-23T03:04:41.9271552Z Generating XML reports...
2022-11-23T03:04:41.9272102Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025513.xml
2022-11-23T03:04:41.9272469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9272691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9273083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9273273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9273528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7imdstv
2022-11-23T03:04:41.9273799Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7imdstv/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9273818Z 
2022-11-23T03:04:41.9273926Z Running tests...
2022-11-23T03:04:41.9274191Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9274547Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9274850Z   test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9275054Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92003
2022-11-23T03:04:41.9275275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92004
2022-11-23T03:04:41.9275485Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92005
2022-11-23T03:04:41.9275697Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92006
2022-11-23T03:04:41.9276070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9276245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9276722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9276914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9277267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9277440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9277809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9277996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9278354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9278528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9278904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9279090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9279451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9279608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9279984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9280170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9280429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdw_yqem
2022-11-23T03:04:41.9280699Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdw_yqem/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9280956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4y5o6ng8
2022-11-23T03:04:41.9281222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4y5o6ng8/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9281473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfc76wxk
2022-11-23T03:04:41.9281762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfc76wxk/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9282020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_f1o4aci
2022-11-23T03:04:41.9282282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_f1o4aci/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9282511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9282735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9282959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9283189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9283336Z fi_getinfo: -61
2022-11-23T03:04:41.9283455Z fi_getinfo: -61
2022-11-23T03:04:41.9283590Z fi_getinfo: -61
2022-11-23T03:04:41.9283726Z fi_getinfo: -61
2022-11-23T03:04:41.9283823Z ok (10.445s)
2022-11-23T03:04:41.9283845Z 
2022-11-23T03:04:41.9284108Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9284221Z Ran 1 test in 10.446s
2022-11-23T03:04:41.9284241Z 
2022-11-23T03:04:41.9284332Z OK
2022-11-23T03:04:41.9284351Z 
2022-11-23T03:04:41.9284472Z Generating XML reports...
2022-11-23T03:04:41.9285001Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025526.xml
2022-11-23T03:04:41.9285372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9285596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9285974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9286166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9286424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdzljim7x
2022-11-23T03:04:41.9286693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdzljim7x/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9286713Z 
2022-11-23T03:04:41.9286820Z Running tests...
2022-11-23T03:04:41.9287064Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9287425Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9287753Z   test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9287976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92362
2022-11-23T03:04:41.9288192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92363
2022-11-23T03:04:41.9288403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92364
2022-11-23T03:04:41.9288620Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92365
2022-11-23T03:04:41.9288994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9289168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9289527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9289715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9290082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9290252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9290625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9290859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9291228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9291404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9291772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9291939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9292292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9292467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9292840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9293025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9293278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm18d9wvk
2022-11-23T03:04:41.9293547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm18d9wvk/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9293796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptt_6jp_v
2022-11-23T03:04:41.9294044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptt_6jp_v/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9294294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8c0uayw6
2022-11-23T03:04:41.9294607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8c0uayw6/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9294860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkrgg28d
2022-11-23T03:04:41.9295131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkrgg28d/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9295362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9295589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9295817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9296040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9296171Z fi_getinfo: -61
2022-11-23T03:04:41.9296304Z fi_getinfo: -61
2022-11-23T03:04:41.9296441Z fi_getinfo: -61
2022-11-23T03:04:41.9296583Z fi_getinfo: -61
2022-11-23T03:04:41.9296683Z ok (4.549s)
2022-11-23T03:04:41.9296703Z 
2022-11-23T03:04:41.9296965Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9297072Z Ran 1 test in 4.549s
2022-11-23T03:04:41.9297092Z 
2022-11-23T03:04:41.9297165Z OK
2022-11-23T03:04:41.9297183Z 
2022-11-23T03:04:41.9297303Z Generating XML reports...
2022-11-23T03:04:41.9297856Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025539.xml
2022-11-23T03:04:41.9298225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9298404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9298785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9298977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9299232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqd_fb4h
2022-11-23T03:04:41.9299482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqd_fb4h/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9299520Z 
2022-11-23T03:04:41.9299610Z Running tests...
2022-11-23T03:04:41.9299921Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9300290Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9300614Z   test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9300835Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92549
2022-11-23T03:04:41.9301055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92550
2022-11-23T03:04:41.9301274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92551
2022-11-23T03:04:41.9301490Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92552
2022-11-23T03:04:41.9301846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9302026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9302403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9302591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9302955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9303127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9303498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9303740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9304109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9304261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9304641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9304830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9305191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9305361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9305731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9305921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9306179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97pvfw9g
2022-11-23T03:04:41.9306433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97pvfw9g/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9306689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7w13sf3d
2022-11-23T03:04:41.9307139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7w13sf3d/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9307402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbd11a4c2
2022-11-23T03:04:41.9307667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbd11a4c2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9307912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt38hcz4g
2022-11-23T03:04:41.9322339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt38hcz4g/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9322647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9322879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9323251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9323492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9323665Z fi_getinfo: -61
2022-11-23T03:04:41.9323805Z fi_getinfo: -61
2022-11-23T03:04:41.9323940Z fi_getinfo: -61
2022-11-23T03:04:41.9324076Z fi_getinfo: -61
2022-11-23T03:04:41.9324175Z ok (4.503s)
2022-11-23T03:04:41.9324199Z 
2022-11-23T03:04:41.9324467Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9324581Z Ran 1 test in 4.503s
2022-11-23T03:04:41.9324600Z 
2022-11-23T03:04:41.9324676Z OK
2022-11-23T03:04:41.9324699Z 
2022-11-23T03:04:41.9324827Z Generating XML reports...
2022-11-23T03:04:41.9325384Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025547.xml
2022-11-23T03:04:41.9325764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9325946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9326331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9326525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9326782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfb4e7kno
2022-11-23T03:04:41.9327054Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfb4e7kno/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9327150Z 
2022-11-23T03:04:41.9327248Z Running tests...
2022-11-23T03:04:41.9327517Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9327877Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9328200Z   test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9328420Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92736
2022-11-23T03:04:41.9328637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92737
2022-11-23T03:04:41.9328853Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92738
2022-11-23T03:04:41.9329070Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92739
2022-11-23T03:04:41.9329427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9329607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9329988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9330178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9330551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9330726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9331098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9331286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9331645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9331803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9332179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9332365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9332784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9332963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9333344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9333536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9333796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2igk3w6
2022-11-23T03:04:41.9334049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2igk3w6/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9334309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqffy0yis
2022-11-23T03:04:41.9334578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqffy0yis/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9334832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4wyz_fq
2022-11-23T03:04:41.9335101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4wyz_fq/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9335352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwh2lad03
2022-11-23T03:04:41.9335618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwh2lad03/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9335848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9336072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9336332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9336559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9336708Z fi_getinfo: -61
2022-11-23T03:04:41.9336846Z fi_getinfo: -61
2022-11-23T03:04:41.9336982Z fi_getinfo: -61
2022-11-23T03:04:41.9337123Z fi_getinfo: -61
2022-11-23T03:04:41.9337223Z ok (4.551s)
2022-11-23T03:04:41.9337244Z 
2022-11-23T03:04:41.9337492Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9337608Z Ran 1 test in 4.551s
2022-11-23T03:04:41.9337627Z 
2022-11-23T03:04:41.9337719Z OK
2022-11-23T03:04:41.9337738Z 
2022-11-23T03:04:41.9337861Z Generating XML reports...
2022-11-23T03:04:41.9338410Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025554.xml
2022-11-23T03:04:41.9338784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9338966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9339350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9339545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9339782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplu571uy8
2022-11-23T03:04:41.9340052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplu571uy8/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9340071Z 
2022-11-23T03:04:41.9340179Z Running tests...
2022-11-23T03:04:41.9340443Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9340800Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9341104Z   test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9341327Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92911
2022-11-23T03:04:41.9341543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92912
2022-11-23T03:04:41.9341785Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92913
2022-11-23T03:04:41.9342009Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92914
2022-11-23T03:04:41.9342384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9342561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9342938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9343130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9343500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9343671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9344044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9344212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9344574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9344748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9345117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9345302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9345719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9345896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9346272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9346445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9346701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gahvs2w
2022-11-23T03:04:41.9347274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gahvs2w/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9347540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2ag5jby
2022-11-23T03:04:41.9347808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2ag5jby/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9348063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp01krq6rz
2022-11-23T03:04:41.9348331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp01krq6rz/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9348584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxba4ekxf
2022-11-23T03:04:41.9348853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxba4ekxf/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9349065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9349287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9349515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9349740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9349894Z fi_getinfo: -61
2022-11-23T03:04:41.9350035Z fi_getinfo: -61
2022-11-23T03:04:41.9350175Z fi_getinfo: -61
2022-11-23T03:04:41.9350293Z fi_getinfo: -61
2022-11-23T03:04:41.9350393Z ok (4.541s)
2022-11-23T03:04:41.9350414Z 
2022-11-23T03:04:41.9350677Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9350788Z Ran 1 test in 4.541s
2022-11-23T03:04:41.9350807Z 
2022-11-23T03:04:41.9350898Z OK
2022-11-23T03:04:41.9350917Z 
2022-11-23T03:04:41.9351133Z Generating XML reports...
2022-11-23T03:04:41.9351698Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025601.xml
2022-11-23T03:04:41.9352069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9352226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9352606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9352804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9353059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpea4mmonv
2022-11-23T03:04:41.9353327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpea4mmonv/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9353347Z 
2022-11-23T03:04:41.9353457Z Running tests...
2022-11-23T03:04:41.9353719Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9354078Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9354386Z   test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9354586Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93098
2022-11-23T03:04:41.9354802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93099
2022-11-23T03:04:41.9355089Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93100
2022-11-23T03:04:41.9355306Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93101
2022-11-23T03:04:41.9355679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9355859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9356239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9356430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9356792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9356948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9357319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9357509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9357868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9358045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9358415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9358602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9358958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9359111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9359486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9359676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9359931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvyurdck0
2022-11-23T03:04:41.9360202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvyurdck0/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9360500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qcyint5
2022-11-23T03:04:41.9360773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qcyint5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9361030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqtulr0z
2022-11-23T03:04:41.9361301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqtulr0z/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9361536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_o0xl_q
2022-11-23T03:04:41.9361808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_o0xl_q/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9362037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9362257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9362485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9362709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9362856Z fi_getinfo: -61
2022-11-23T03:04:41.9362995Z fi_getinfo: -61
2022-11-23T03:04:41.9363112Z fi_getinfo: -61
2022-11-23T03:04:41.9363247Z fi_getinfo: -61
2022-11-23T03:04:41.9363347Z ok (6.772s)
2022-11-23T03:04:41.9363367Z 
2022-11-23T03:04:41.9363630Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9363743Z Ran 1 test in 6.772s
2022-11-23T03:04:41.9363762Z 
2022-11-23T03:04:41.9363909Z OK
2022-11-23T03:04:41.9363928Z 
2022-11-23T03:04:41.9364052Z Generating XML reports...
2022-11-23T03:04:41.9364587Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025608.xml
2022-11-23T03:04:41.9364961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9365139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9365516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9365708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9365963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6aps1ic5
2022-11-23T03:04:41.9366231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6aps1ic5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9366255Z 
2022-11-23T03:04:41.9366362Z Running tests...
2022-11-23T03:04:41.9366623Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9366965Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9367287Z   test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9367505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93445
2022-11-23T03:04:41.9367721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93446
2022-11-23T03:04:41.9367936Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93447
2022-11-23T03:04:41.9368150Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93448
2022-11-23T03:04:41.9368519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9368697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9369059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9369250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9369669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9369846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9370221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9370409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9370769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9370946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9371323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9371494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9371860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9372035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9372404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9372592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9372851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxqjgb3p
2022-11-23T03:04:41.9373123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxqjgb3p/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9373430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4568opia
2022-11-23T03:04:41.9373698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4568opia/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9373935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z4wzta9
2022-11-23T03:04:41.9374201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z4wzta9/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9374451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4siwha49
2022-11-23T03:04:41.9374713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4siwha49/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9374940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9375163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9375393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9375618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9375749Z fi_getinfo: -61
2022-11-23T03:04:41.9375887Z fi_getinfo: -61
2022-11-23T03:04:41.9376022Z fi_getinfo: -61
2022-11-23T03:04:41.9376162Z fi_getinfo: -61
2022-11-23T03:04:41.9376262Z ok (6.734s)
2022-11-23T03:04:41.9376282Z 
2022-11-23T03:04:41.9376548Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9376660Z Ran 1 test in 6.734s
2022-11-23T03:04:41.9376680Z 
2022-11-23T03:04:41.9376753Z OK
2022-11-23T03:04:41.9376789Z 
2022-11-23T03:04:41.9376894Z Generating XML reports...
2022-11-23T03:04:41.9377438Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025618.xml
2022-11-23T03:04:41.9377815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9377992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9378369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9378607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9378868Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqdl2nl1
2022-11-23T03:04:41.9379137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqdl2nl1/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9379157Z 
2022-11-23T03:04:41.9379246Z Running tests...
2022-11-23T03:04:41.9379512Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9379873Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9380202Z   test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9380420Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93792
2022-11-23T03:04:41.9380638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93793
2022-11-23T03:04:41.9380855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93794
2022-11-23T03:04:41.9381070Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93795
2022-11-23T03:04:41.9381428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9381605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9381985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9382175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9382612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9382785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9383162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9383351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9383712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9383867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9384233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9384419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9384779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9384951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9385323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9385513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9385770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc72wao83
2022-11-23T03:04:41.9386038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc72wao83/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9386271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4dlcn5gs
2022-11-23T03:04:41.9386538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4dlcn5gs/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9387084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5qpgiuf
2022-11-23T03:04:41.9387351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5qpgiuf/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9387599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3am4qtq
2022-11-23T03:04:41.9387947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3am4qtq/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9388185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9388409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9388617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9388842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9389000Z fi_getinfo: -61
2022-11-23T03:04:41.9389143Z fi_getinfo: -61
2022-11-23T03:04:41.9389281Z fi_getinfo: -61
2022-11-23T03:04:41.9389415Z fi_getinfo: -61
2022-11-23T03:04:41.9389514Z ok (6.597s)
2022-11-23T03:04:41.9389534Z 
2022-11-23T03:04:41.9389779Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9389891Z Ran 1 test in 6.597s
2022-11-23T03:04:41.9389910Z 
2022-11-23T03:04:41.9390001Z OK
2022-11-23T03:04:41.9390024Z 
2022-11-23T03:04:41.9390147Z Generating XML reports...
2022-11-23T03:04:41.9390697Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025627.xml
2022-11-23T03:04:41.9391070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9391246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9391623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9391899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9392137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq06mmbr9
2022-11-23T03:04:41.9392406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq06mmbr9/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9392429Z 
2022-11-23T03:04:41.9392538Z Running tests...
2022-11-23T03:04:41.9392806Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9393166Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9393486Z   test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9393706Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94139
2022-11-23T03:04:41.9393925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94140
2022-11-23T03:04:41.9394127Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94141
2022-11-23T03:04:41.9394342Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94142
2022-11-23T03:04:41.9394718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9394893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9395274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9395465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9395832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9396005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9396385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9396554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9396916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9397153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9397535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9397722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9398083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9398259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9398633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9398825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9399064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5kzq_lg
2022-11-23T03:04:41.9399339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5kzq_lg/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9399595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvo9dbe_p
2022-11-23T03:04:41.9399867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvo9dbe_p/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9400120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpveeb4v78
2022-11-23T03:04:41.9400383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpveeb4v78/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9400636Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplzplkpw1
2022-11-23T03:04:41.9400959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplzplkpw1/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9401171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9401395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9401625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9401852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9401998Z fi_getinfo: -61
2022-11-23T03:04:41.9402133Z fi_getinfo: -61
2022-11-23T03:04:41.9402269Z fi_getinfo: -61
2022-11-23T03:04:41.9402407Z fi_getinfo: -61
2022-11-23T03:04:41.9402488Z ok (6.542s)
2022-11-23T03:04:41.9402508Z 
2022-11-23T03:04:41.9402771Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9402883Z Ran 1 test in 6.542s
2022-11-23T03:04:41.9402905Z 
2022-11-23T03:04:41.9402997Z OK
2022-11-23T03:04:41.9403016Z 
2022-11-23T03:04:41.9403137Z Generating XML reports...
2022-11-23T03:04:41.9403686Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025636.xml
2022-11-23T03:04:41.9404063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9404239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9404602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9404793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9405047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphi75v65c
2022-11-23T03:04:41.9405316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphi75v65c/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9405339Z 
2022-11-23T03:04:41.9405447Z Running tests...
2022-11-23T03:04:41.9405709Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9406067Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9406450Z   test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9406677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94486
2022-11-23T03:04:41.9406875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94487
2022-11-23T03:04:41.9407089Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94488
2022-11-23T03:04:41.9407305Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94489
2022-11-23T03:04:41.9407680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9407859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9408238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9408433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9408796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9408954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9409331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9409523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9409884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9410106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9410483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9410670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9411076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9411250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9411608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9411796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9412055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdqbrktm
2022-11-23T03:04:41.9412330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdqbrktm/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9412583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67gq3o58
2022-11-23T03:04:41.9412848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67gq3o58/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9413102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0b3dq4ty
2022-11-23T03:04:41.9413365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0b3dq4ty/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9413599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyk4pr8wv
2022-11-23T03:04:41.9413863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyk4pr8wv/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9414089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9414312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9414543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9414693Z fi_getinfo: -61
2022-11-23T03:04:41.9414830Z fi_getinfo: -61
2022-11-23T03:04:41.9414966Z fi_getinfo: -61
2022-11-23T03:04:41.9415227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9415375Z fi_getinfo: -61
2022-11-23T03:04:41.9415475Z ok (6.856s)
2022-11-23T03:04:41.9415495Z 
2022-11-23T03:04:41.9415758Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9415869Z Ran 1 test in 6.856s
2022-11-23T03:04:41.9415888Z 
2022-11-23T03:04:41.9415981Z OK
2022-11-23T03:04:41.9415999Z 
2022-11-23T03:04:41.9416123Z Generating XML reports...
2022-11-23T03:04:41.9416670Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025646.xml
2022-11-23T03:04:41.9417029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9417204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9417587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9417778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9418033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1kgf587
2022-11-23T03:04:41.9418299Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1kgf587/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9418318Z 
2022-11-23T03:04:41.9418426Z Running tests...
2022-11-23T03:04:41.9418685Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9419027Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9419408Z   test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9419627Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94833
2022-11-23T03:04:41.9419849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94834
2022-11-23T03:04:41.9420063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94835
2022-11-23T03:04:41.9420278Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94836
2022-11-23T03:04:41.9420649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9420823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9421202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9421378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9421748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9421920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9422295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9422482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9422843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9423018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9423387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9423579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9423921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9424093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9424512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9424706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9424965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxnq7vik
2022-11-23T03:04:41.9425236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxnq7vik/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9425490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6n2ftf0h
2022-11-23T03:04:41.9425758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6n2ftf0h/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9425998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqoqnve_
2022-11-23T03:04:41.9426264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqoqnve_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9426517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2a56a1sr
2022-11-23T03:04:41.9426953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2a56a1sr/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9427203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9427419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9427649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9427876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9428118Z fi_getinfo: -61
2022-11-23T03:04:41.9428237Z fi_getinfo: -61
2022-11-23T03:04:41.9428372Z fi_getinfo: -61
2022-11-23T03:04:41.9428507Z fi_getinfo: -61
2022-11-23T03:04:41.9428607Z ok (6.753s)
2022-11-23T03:04:41.9428627Z 
2022-11-23T03:04:41.9428894Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9429006Z Ran 1 test in 6.754s
2022-11-23T03:04:41.9429030Z 
2022-11-23T03:04:41.9429122Z OK
2022-11-23T03:04:41.9429141Z 
2022-11-23T03:04:41.9429246Z Generating XML reports...
2022-11-23T03:04:41.9429795Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025655.xml
2022-11-23T03:04:41.9430163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9430338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9430718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9430914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9431169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8_5bybs
2022-11-23T03:04:41.9431436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8_5bybs/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9431456Z 
2022-11-23T03:04:41.9431564Z Running tests...
2022-11-23T03:04:41.9431812Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9432176Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9432518Z   test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9432724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95180
2022-11-23T03:04:41.9432946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95181
2022-11-23T03:04:41.9433161Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95182
2022-11-23T03:04:41.9433377Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95183
2022-11-23T03:04:41.9433867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9434033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9434414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9434601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9434970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9435143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9435522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9435710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9436076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9436253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9436602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9436789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9437145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9437316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9437751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9437937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9438195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt3t8ij61
2022-11-23T03:04:41.9438465Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt3t8ij61/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9438702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfem6kzr
2022-11-23T03:04:41.9438972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfem6kzr/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9439223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1cj60lx7
2022-11-23T03:04:41.9439485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1cj60lx7/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9439737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpci9bl_t6
2022-11-23T03:04:41.9439997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpci9bl_t6/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9440224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9440448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9440675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9440884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9441032Z fi_getinfo: -61
2022-11-23T03:04:41.9441169Z fi_getinfo: -61
2022-11-23T03:04:41.9441304Z fi_getinfo: -61
2022-11-23T03:04:41.9441438Z fi_getinfo: -61
2022-11-23T03:04:41.9441537Z ok (6.728s)
2022-11-23T03:04:41.9441557Z 
2022-11-23T03:04:41.9441819Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9441916Z Ran 1 test in 6.728s
2022-11-23T03:04:41.9441935Z 
2022-11-23T03:04:41.9442027Z OK
2022-11-23T03:04:41.9442046Z 
2022-11-23T03:04:41.9442168Z Generating XML reports...
2022-11-23T03:04:41.9442773Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025705.xml
2022-11-23T03:04:41.9443144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9443322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9443699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9443888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9444120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp00cvbsbu
2022-11-23T03:04:41.9444392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp00cvbsbu/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9444411Z 
2022-11-23T03:04:41.9444519Z Running tests...
2022-11-23T03:04:41.9444781Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9445141Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9445442Z   test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9445659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95527
2022-11-23T03:04:41.9445878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95528
2022-11-23T03:04:41.9446091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95529
2022-11-23T03:04:41.9446285Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95530
2022-11-23T03:04:41.9446714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9446888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9447267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9447465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9447835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9448008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9448373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9448527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9448902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9449096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9449469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9449660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9450022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9450195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9450568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9450756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9450996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7ak8vqv
2022-11-23T03:04:41.9451269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7ak8vqv/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9451522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ovdqa9q
2022-11-23T03:04:41.9451839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ovdqa9q/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9452096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp21n92aew
2022-11-23T03:04:41.9452360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp21n92aew/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9452640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedqi02e3
2022-11-23T03:04:41.9453096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedqi02e3/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9453490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9453715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9453958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9454166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9454353Z fi_getinfo: -61
2022-11-23T03:04:41.9454485Z fi_getinfo: -61
2022-11-23T03:04:41.9454617Z fi_getinfo: -61
2022-11-23T03:04:41.9454753Z fi_getinfo: -61
2022-11-23T03:04:41.9454835Z ok (10.448s)
2022-11-23T03:04:41.9454856Z 
2022-11-23T03:04:41.9455121Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9455234Z Ran 1 test in 10.448s
2022-11-23T03:04:41.9455253Z 
2022-11-23T03:04:41.9455347Z OK
2022-11-23T03:04:41.9455367Z 
2022-11-23T03:04:41.9455491Z Generating XML reports...
2022-11-23T03:04:41.9456036Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025714.xml
2022-11-23T03:04:41.9456517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9456695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9457058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9457252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9457505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxk4m8q45
2022-11-23T03:04:41.9457771Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxk4m8q45/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9457791Z 
2022-11-23T03:04:41.9457899Z Running tests...
2022-11-23T03:04:41.9458162Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9458525Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9458835Z   test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9459056Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95886
2022-11-23T03:04:41.9459259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95887
2022-11-23T03:04:41.9459475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95888
2022-11-23T03:04:41.9459688Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95889
2022-11-23T03:04:41.9460059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9460235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9460616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9460811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9461177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9461332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9461759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9461941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9462321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9462513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9462887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9463079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9463449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9463623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9463983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9464171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9464428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuug5b80c
2022-11-23T03:04:41.9464696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuug5b80c/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9464951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43wrnrxq
2022-11-23T03:04:41.9465217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43wrnrxq/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9465523Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk635ba0u
2022-11-23T03:04:41.9465789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk635ba0u/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9466045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplmi51958
2022-11-23T03:04:41.9466291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplmi51958/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9466520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9466744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9467171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9467391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9467551Z fi_getinfo: -61
2022-11-23T03:04:41.9467686Z fi_getinfo: -61
2022-11-23T03:04:41.9467804Z fi_getinfo: -61
2022-11-23T03:04:41.9467940Z fi_getinfo: -61
2022-11-23T03:04:41.9468040Z ok (10.347s)
2022-11-23T03:04:41.9468061Z 
2022-11-23T03:04:41.9468325Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9468441Z Ran 1 test in 10.348s
2022-11-23T03:04:41.9468461Z 
2022-11-23T03:04:41.9468552Z OK
2022-11-23T03:04:41.9468571Z 
2022-11-23T03:04:41.9468696Z Generating XML reports...
2022-11-23T03:04:41.9469244Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025727.xml
2022-11-23T03:04:41.9469595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9469771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9470158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9470347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9470600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwnarplru
2022-11-23T03:04:41.9470955Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwnarplru/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9470977Z 
2022-11-23T03:04:41.9471090Z Running tests...
2022-11-23T03:04:41.9471360Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9471715Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9472000Z   test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9472218Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96237
2022-11-23T03:04:41.9472439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96238
2022-11-23T03:04:41.9472653Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96239
2022-11-23T03:04:41.9472866Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96240
2022-11-23T03:04:41.9473244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9473420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9473802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9473975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9474343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9474594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9474970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9475160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9475524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9475701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9476071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9476258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9476602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9476774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9477146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9477334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9477590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hg5t5y3
2022-11-23T03:04:41.9477862Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hg5t5y3/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9478115Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqrdoos4_
2022-11-23T03:04:41.9478382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqrdoos4_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9478637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5nqc4wa
2022-11-23T03:04:41.9478885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5nqc4wa/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9479141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7g3ihu2o
2022-11-23T03:04:41.9479410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7g3ihu2o/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9479640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9479911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9480142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9480370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9480515Z fi_getinfo: -61
2022-11-23T03:04:41.9480597Z ok (4.426s)
2022-11-23T03:04:41.9480617Z 
2022-11-23T03:04:41.9480882Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9480994Z Ran 1 test in 4.426s
2022-11-23T03:04:41.9481017Z 
2022-11-23T03:04:41.9481109Z OK
2022-11-23T03:04:41.9481128Z 
2022-11-23T03:04:41.9481251Z Generating XML reports...
2022-11-23T03:04:41.9481797Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025740.xml
2022-11-23T03:04:41.9482172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9482350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9482713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9482909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9483162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8h4kffkq
2022-11-23T03:04:41.9483428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8h4kffkq/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9483495Z 
2022-11-23T03:04:41.9483610Z Running tests...
2022-11-23T03:04:41.9483875Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9484235Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9484541Z   test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9484761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96409
2022-11-23T03:04:41.9484961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96410
2022-11-23T03:04:41.9485179Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96411
2022-11-23T03:04:41.9485396Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96412
2022-11-23T03:04:41.9485771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9485966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9486351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9486520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9486888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9487043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9487417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9487605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9487964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9488143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9488511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9488696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9489100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9489278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9489637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9489827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9490080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp70zx6y74
2022-11-23T03:04:41.9490347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp70zx6y74/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9490603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfu2_8_sh
2022-11-23T03:04:41.9490866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfu2_8_sh/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9491124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwc99vkfk
2022-11-23T03:04:41.9491393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwc99vkfk/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9491630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wndouap
2022-11-23T03:04:41.9491898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wndouap/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9492127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9492350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9492626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9492853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9493002Z fi_getinfo: -61
2022-11-23T03:04:41.9493138Z fi_getinfo: -61
2022-11-23T03:04:41.9493255Z fi_getinfo: -61
2022-11-23T03:04:41.9493395Z fi_getinfo: -61
2022-11-23T03:04:41.9493494Z ok (10.359s)
2022-11-23T03:04:41.9493514Z 
2022-11-23T03:04:41.9493780Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9493893Z Ran 1 test in 10.359s
2022-11-23T03:04:41.9493912Z 
2022-11-23T03:04:41.9494007Z OK
2022-11-23T03:04:41.9494026Z 
2022-11-23T03:04:41.9494149Z Generating XML reports...
2022-11-23T03:04:41.9494696Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025747.xml
2022-11-23T03:04:41.9495046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9495228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9495604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9495801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9496058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkatzawgd
2022-11-23T03:04:41.9496329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkatzawgd/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9496348Z 
2022-11-23T03:04:41.9496456Z Running tests...
2022-11-23T03:04:41.9496718Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9497075Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9497370Z   test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9497589Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96768
2022-11-23T03:04:41.9497809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96769
2022-11-23T03:04:41.9498078Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96770
2022-11-23T03:04:41.9498298Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96771
2022-11-23T03:04:41.9498673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9498851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9499230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9499403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9499774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9499948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9500322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9500511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9500872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9501046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9501420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9501607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9502064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9502236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9502607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9502798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9503054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tv_ah5f
2022-11-23T03:04:41.9503323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tv_ah5f/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9503577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34axza62
2022-11-23T03:04:41.9503842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34axza62/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9504079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwqdx3xhh
2022-11-23T03:04:41.9504347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwqdx3xhh/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9504595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2njm4nfy
2022-11-23T03:04:41.9504861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2njm4nfy/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9505089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9505317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9505537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9505761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9505910Z skip: Need at least 4 CUDA devices (4.413s)
2022-11-23T03:04:41.9505933Z 
2022-11-23T03:04:41.9506187Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9506299Z Ran 1 test in 4.413s
2022-11-23T03:04:41.9506318Z 
2022-11-23T03:04:41.9506426Z OK (skipped=1)
2022-11-23T03:04:41.9506444Z 
2022-11-23T03:04:41.9506567Z Generating XML reports...
2022-11-23T03:04:41.9507357Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025800.xml
2022-11-23T03:04:41.9507756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9507933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9508311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9508503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9508745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprq6xibb9
2022-11-23T03:04:41.9509016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprq6xibb9/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9509035Z 
2022-11-23T03:04:41.9509143Z Running tests...
2022-11-23T03:04:41.9509412Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9509770Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9510085Z   test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9510307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96939
2022-11-23T03:04:41.9510525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96940
2022-11-23T03:04:41.9510722Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96941
2022-11-23T03:04:41.9511046Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96942
2022-11-23T03:04:41.9511427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9511602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9511983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9512176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9512540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9512714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9513083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9513258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9513620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9513791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9514159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9514348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9514708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9514882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9515256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9515426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9515688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1806hvms
2022-11-23T03:04:41.9515956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1806hvms/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9516209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_feum9x
2022-11-23T03:04:41.9516521Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1v4yw57
2022-11-23T03:04:41.9516797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_feum9x/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9517057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1v4yw57/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9517310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1y_pifj
2022-11-23T03:04:41.9517573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1y_pifj/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9517791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9518019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9518246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9518473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9518621Z skip: Need at least 4 CUDA devices (4.330s)
2022-11-23T03:04:41.9518641Z 
2022-11-23T03:04:41.9518909Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9519021Z Ran 1 test in 4.330s
2022-11-23T03:04:41.9519040Z 
2022-11-23T03:04:41.9519148Z OK (skipped=1)
2022-11-23T03:04:41.9519167Z 
2022-11-23T03:04:41.9519270Z Generating XML reports...
2022-11-23T03:04:41.9519823Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025808.xml
2022-11-23T03:04:41.9520253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9520428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9520811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9521005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9521261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzx4elzds
2022-11-23T03:04:41.9521535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzx4elzds/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9521554Z 
2022-11-23T03:04:41.9521663Z Running tests...
2022-11-23T03:04:41.9521907Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9522265Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9522581Z   test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9522800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97110
2022-11-23T03:04:41.9523021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97111
2022-11-23T03:04:41.9523235Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97112
2022-11-23T03:04:41.9523446Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97113
2022-11-23T03:04:41.9523819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9523993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9524353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9524547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9524912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9525085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9525522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9525718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9526079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9526251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9526600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9526794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9527162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9527332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9527700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9527886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9528143Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazkieuo3
2022-11-23T03:04:41.9528415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazkieuo3/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9528669Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3g94p58t
2022-11-23T03:04:41.9528916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3g94p58t/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9529226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wyhxc25
2022-11-23T03:04:41.9529493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wyhxc25/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9529750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqqnz6bsy
2022-11-23T03:04:41.9530017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqqnz6bsy/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9530246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9530474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9530702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9530906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9531059Z fi_getinfo: -61
2022-11-23T03:04:41.9531195Z fi_getinfo: -61
2022-11-23T03:04:41.9531333Z fi_getinfo: -61
2022-11-23T03:04:41.9531468Z fi_getinfo: -61
2022-11-23T03:04:41.9531569Z ok (4.628s)
2022-11-23T03:04:41.9531588Z 
2022-11-23T03:04:41.9531852Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9531949Z Ran 1 test in 4.628s
2022-11-23T03:04:41.9531987Z 
2022-11-23T03:04:41.9532062Z OK
2022-11-23T03:04:41.9532080Z 
2022-11-23T03:04:41.9532204Z Generating XML reports...
2022-11-23T03:04:41.9532751Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025815.xml
2022-11-23T03:04:41.9533124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9533300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9533679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9533875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9534130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcplkw52
2022-11-23T03:04:41.9534433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcplkw52/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9534473Z 
2022-11-23T03:04:41.9534566Z Running tests...
2022-11-23T03:04:41.9534829Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9535188Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9535485Z   test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9535704Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97297
2022-11-23T03:04:41.9535928Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97298
2022-11-23T03:04:41.9536143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97299
2022-11-23T03:04:41.9536355Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97300
2022-11-23T03:04:41.9536715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9536890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9537268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9537459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9537824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9537998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9538427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9538618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9538964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9539138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9539505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9539691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9540046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9540217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9540598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9540783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9541039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl91fmcq1
2022-11-23T03:04:41.9541292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl91fmcq1/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9541546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppkemo7lx
2022-11-23T03:04:41.9541814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppkemo7lx/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9542068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobewo1gq
2022-11-23T03:04:41.9542334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobewo1gq/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9542590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfr6knzx2
2022-11-23T03:04:41.9542855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfr6knzx2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9543086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9543342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9543576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9543802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9543950Z fi_getinfo: -61
2022-11-23T03:04:41.9544092Z fi_getinfo: -61
2022-11-23T03:04:41.9544229Z fi_getinfo: -61
2022-11-23T03:04:41.9544366Z fi_getinfo: -61
2022-11-23T03:04:41.9544481Z On WorkerInfo(id=1, name=worker1):
2022-11-23T03:04:41.9557264Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f4fde8f859b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f4fde8f3dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7f4feab3f13d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f4feab4067f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f4feab41ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f4feae36b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2a0be3e (0x7f4fe17aee3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2a0bf46 (0x7f4fe17aef46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f4feb948c58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x35f0340 (0x7f4fed1bb340 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x35f0ab9 (0x7f4fed1bbab9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f4feb982e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2ff562 (0x7f4ff68a9562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2ff956 (0x7f4ff68a9956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x5559d5d6cc68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x5559d5d28499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x5559d5d285fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x5559d5cd44b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x5559d5d71098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x5559d5d1e742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x5559d5cd6faa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x5559d5d72774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x5559d5d1e742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x5559d5cd6faa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xaa8dba (0x7f4ff7052dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f4ff7050ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f4ff70542d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f4ff7055b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f4fee5f653c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f4ff70540c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4a249c3 (0x7f4fee5ef9c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f4fee5f0558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f4fee5ea857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x4a54542 (0x7f4fee61f542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f4fde8e690b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7f500e74cbf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7f502eda06db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7f502eac961f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T03:04:41.9557572Z Traceback (most recent call last):
2022-11-23T03:04:41.9557956Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T03:04:41.9558149Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T03:04:41.9558559Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2022-11-23T03:04:41.9558681Z     return x.cpu() + y.cuda()
2022-11-23T03:04:41.9558929Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-11-23T03:04:41.9559197Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-11-23T03:04:41.9559763Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f4fde8f859b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.9560390Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f4fde8f3dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.9560940Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7f4feab3f13d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9561421Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f4feab4067f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9561995Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f4feab41ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9562514Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f4feae36b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9562909Z frame #6: <unknown function> + 0x2a0be3e (0x7f4fe17aee3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.9563292Z frame #7: <unknown function> + 0x2a0bf46 (0x7f4fe17aef46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.9563849Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f4feb948c58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9564229Z frame #9: <unknown function> + 0x35f0340 (0x7f4fed1bb340 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9564609Z frame #10: <unknown function> + 0x35f0ab9 (0x7f4fed1bbab9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9565172Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f4feb982e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9565539Z frame #12: <unknown function> + 0x2ff562 (0x7f4ff68a9562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9565925Z frame #13: <unknown function> + 0x2ff956 (0x7f4ff68a9956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9566124Z frame #14: <unknown function> + 0x1ddc68 (0x5559d5d6cc68 in /opt/conda/bin/python)
2022-11-23T03:04:41.9566314Z frame #15: <unknown function> + 0x199499 (0x5559d5d28499 in /opt/conda/bin/python)
2022-11-23T03:04:41.9566501Z frame #16: <unknown function> + 0x1995fa (0x5559d5d285fa in /opt/conda/bin/python)
2022-11-23T03:04:41.9566688Z frame #17: PyNumber_Add + 0x41 (0x5559d5cd44b1 in /opt/conda/bin/python)
2022-11-23T03:04:41.9566893Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x5559d5d71098 in /opt/conda/bin/python)
2022-11-23T03:04:41.9567087Z frame #19: <unknown function> + 0x18f742 (0x5559d5d1e742 in /opt/conda/bin/python)
2022-11-23T03:04:41.9567273Z frame #20: _PyObject_Call + 0x20a (0x5559d5cd6faa in /opt/conda/bin/python)
2022-11-23T03:04:41.9567461Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x5559d5d72774 in /opt/conda/bin/python)
2022-11-23T03:04:41.9567651Z frame #22: <unknown function> + 0x18f742 (0x5559d5d1e742 in /opt/conda/bin/python)
2022-11-23T03:04:41.9567837Z frame #23: _PyObject_Call + 0x20a (0x5559d5cd6faa in /opt/conda/bin/python)
2022-11-23T03:04:41.9568230Z frame #24: <unknown function> + 0xaa8dba (0x7f4ff7052dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9568751Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f4ff7050ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9569413Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f4ff70542d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9570144Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f4ff7055b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9570923Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f4fee5f653c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9571703Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f4ff70540c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9572091Z frame #30: <unknown function> + 0x4a249c3 (0x7f4fee5ef9c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9572761Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f4fee5f0558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9573392Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f4fee5ea857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9573773Z frame #33: <unknown function> + 0x4a54542 (0x7f4fee61f542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9574307Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f4fde8e690b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.9574515Z frame #35: <unknown function> + 0xdbbf4 (0x7f500e74cbf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T03:04:41.9574841Z frame #36: <unknown function> + 0x76db (0x7f502eda06db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T03:04:41.9575110Z frame #37: clone + 0x3f (0x7f502eac961f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T03:04:41.9575134Z 
2022-11-23T03:04:41.9575152Z 
2022-11-23T03:04:41.9575287Z On WorkerInfo(id=0, name=worker0):
2022-11-23T03:04:41.9587604Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f163557b59b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f1635576dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7f16417c213d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f16417c367f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f16417c4ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f1641ab9b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2a0be3e (0x7f1638431e3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2a0bf46 (0x7f1638431f46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f16425cbc58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x35f0340 (0x7f1643e3e340 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x35f0ab9 (0x7f1643e3eab9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f1642605e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2ff562 (0x7f164d52c562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2ff956 (0x7f164d52c956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x55c2fca9ac68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x55c2fca56499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x55c2fca565fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x55c2fca024b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x55c2fca9f098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x55c2fca4c742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x55c2fca04faa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55c2fcaa0774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x55c2fca4c742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x55c2fca04faa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xaa8dba (0x7f164dcd5dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f164dcd3ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f164dcd72d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f164dcd8b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f164527953c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f164dcd70c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4a249c3 (0x7f16452729c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f1645273558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f164526d857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x4a54542 (0x7f16452a2542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f163556990b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7f16653cfbf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7f1685a236db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7f168574c61f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T03:04:41.9587887Z Traceback (most recent call last):
2022-11-23T03:04:41.9588263Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T03:04:41.9588458Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T03:04:41.9588844Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2022-11-23T03:04:41.9588964Z     return x.cpu() + y.cuda()
2022-11-23T03:04:41.9589215Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-11-23T03:04:41.9589496Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-11-23T03:04:41.9590058Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f163557b59b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.9590676Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f1635576dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.9591238Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7f16417c213d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9591702Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f16417c367f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9592276Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f16417c4ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9592793Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f1641ab9b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9593177Z frame #6: <unknown function> + 0x2a0be3e (0x7f1638431e3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.9593552Z frame #7: <unknown function> + 0x2a0bf46 (0x7f1638431f46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.9594089Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f16425cbc58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9594466Z frame #9: <unknown function> + 0x35f0340 (0x7f1643e3e340 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9594842Z frame #10: <unknown function> + 0x35f0ab9 (0x7f1643e3eab9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9595332Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f1642605e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9595718Z frame #12: <unknown function> + 0x2ff562 (0x7f164d52c562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9596099Z frame #13: <unknown function> + 0x2ff956 (0x7f164d52c956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9596345Z frame #14: <unknown function> + 0x1ddc68 (0x55c2fca9ac68 in /opt/conda/bin/python)
2022-11-23T03:04:41.9596544Z frame #15: <unknown function> + 0x199499 (0x55c2fca56499 in /opt/conda/bin/python)
2022-11-23T03:04:41.9596732Z frame #16: <unknown function> + 0x1995fa (0x55c2fca565fa in /opt/conda/bin/python)
2022-11-23T03:04:41.9596917Z frame #17: PyNumber_Add + 0x41 (0x55c2fca024b1 in /opt/conda/bin/python)
2022-11-23T03:04:41.9597107Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x55c2fca9f098 in /opt/conda/bin/python)
2022-11-23T03:04:41.9597299Z frame #19: <unknown function> + 0x18f742 (0x55c2fca4c742 in /opt/conda/bin/python)
2022-11-23T03:04:41.9597488Z frame #20: _PyObject_Call + 0x20a (0x55c2fca04faa in /opt/conda/bin/python)
2022-11-23T03:04:41.9597696Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55c2fcaa0774 in /opt/conda/bin/python)
2022-11-23T03:04:41.9597882Z frame #22: <unknown function> + 0x18f742 (0x55c2fca4c742 in /opt/conda/bin/python)
2022-11-23T03:04:41.9598073Z frame #23: _PyObject_Call + 0x20a (0x55c2fca04faa in /opt/conda/bin/python)
2022-11-23T03:04:41.9598471Z frame #24: <unknown function> + 0xaa8dba (0x7f164dcd5dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9598993Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f164dcd3ffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9599651Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f164dcd72d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9600384Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f164dcd8b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9601148Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f164527953c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9601927Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f164dcd70c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9602310Z frame #30: <unknown function> + 0x4a249c3 (0x7f16452729c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9602973Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f1645273558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9603596Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f164526d857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9603973Z frame #33: <unknown function> + 0x4a54542 (0x7f16452a2542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9604383Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f163556990b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.9604605Z frame #35: <unknown function> + 0xdbbf4 (0x7f16653cfbf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T03:04:41.9604922Z frame #36: <unknown function> + 0x76db (0x7f1685a236db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T03:04:41.9605238Z frame #37: clone + 0x3f (0x7f168574c61f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T03:04:41.9605261Z 
2022-11-23T03:04:41.9605280Z 
2022-11-23T03:04:41.9605417Z On WorkerInfo(id=3, name=worker3):
2022-11-23T03:04:41.9617595Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f0d7ab6759b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f0d7ab62dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7f0d86dae13d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f0d86daf67f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f0d86db0ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f0d870a5b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2a0be3e (0x7f0d7da1de3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2a0bf46 (0x7f0d7da1df46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f0d87bb7c58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x35f0340 (0x7f0d8942a340 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x35f0ab9 (0x7f0d8942aab9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f0d87bf1e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2ff562 (0x7f0d92b18562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2ff956 (0x7f0d92b18956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x55dca7bb0c68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x55dca7b6c499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x55dca7b6c5fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x55dca7b184b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x55dca7bb5098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x55dca7b62742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x55dca7b1afaa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55dca7bb6774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x55dca7b62742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x55dca7b1afaa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xaa8dba (0x7f0d932c1dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f0d932bfffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f0d932c32d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f0d932c4b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f0d8a86553c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f0d932c30c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4a249c3 (0x7f0d8a85e9c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f0d8a85f558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f0d8a859857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x4a54542 (0x7f0d8a88e542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f0d7ab5590b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7f0daa9bbbf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7f0dcb00f6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7f0dcad3861f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T03:04:41.9617855Z Traceback (most recent call last):
2022-11-23T03:04:41.9618223Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T03:04:41.9618414Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T03:04:41.9618816Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2022-11-23T03:04:41.9618940Z     return x.cpu() + y.cuda()
2022-11-23T03:04:41.9619189Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-11-23T03:04:41.9619473Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-11-23T03:04:41.9620033Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f0d7ab6759b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.9620653Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f0d7ab62dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.9621149Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7f0d86dae13d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9621603Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7f0d86daf67f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9622226Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7f0d86db0ef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9622759Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7f0d870a5b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9623146Z frame #6: <unknown function> + 0x2a0be3e (0x7f0d7da1de3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.9623526Z frame #7: <unknown function> + 0x2a0bf46 (0x7f0d7da1df46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.9624082Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7f0d87bb7c58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9624458Z frame #9: <unknown function> + 0x35f0340 (0x7f0d8942a340 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9624832Z frame #10: <unknown function> + 0x35f0ab9 (0x7f0d8942aab9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9625326Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7f0d87bf1e62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9625706Z frame #12: <unknown function> + 0x2ff562 (0x7f0d92b18562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9626149Z frame #13: <unknown function> + 0x2ff956 (0x7f0d92b18956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9626348Z frame #14: <unknown function> + 0x1ddc68 (0x55dca7bb0c68 in /opt/conda/bin/python)
2022-11-23T03:04:41.9626524Z frame #15: <unknown function> + 0x199499 (0x55dca7b6c499 in /opt/conda/bin/python)
2022-11-23T03:04:41.9626719Z frame #16: <unknown function> + 0x1995fa (0x55dca7b6c5fa in /opt/conda/bin/python)
2022-11-23T03:04:41.9627087Z frame #17: PyNumber_Add + 0x41 (0x55dca7b184b1 in /opt/conda/bin/python)
2022-11-23T03:04:41.9627297Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x55dca7bb5098 in /opt/conda/bin/python)
2022-11-23T03:04:41.9627499Z frame #19: <unknown function> + 0x18f742 (0x55dca7b62742 in /opt/conda/bin/python)
2022-11-23T03:04:41.9627681Z frame #20: _PyObject_Call + 0x20a (0x55dca7b1afaa in /opt/conda/bin/python)
2022-11-23T03:04:41.9627891Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x55dca7bb6774 in /opt/conda/bin/python)
2022-11-23T03:04:41.9628083Z frame #22: <unknown function> + 0x18f742 (0x55dca7b62742 in /opt/conda/bin/python)
2022-11-23T03:04:41.9628253Z frame #23: _PyObject_Call + 0x20a (0x55dca7b1afaa in /opt/conda/bin/python)
2022-11-23T03:04:41.9628656Z frame #24: <unknown function> + 0xaa8dba (0x7f0d932c1dba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9629185Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7f0d932bfffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9629846Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7f0d932c32d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9630529Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7f0d932c4b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9631373Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7f0d8a86553c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9632171Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f0d932c30c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9632551Z frame #30: <unknown function> + 0x4a249c3 (0x7f0d8a85e9c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9633222Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7f0d8a85f558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9633847Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f0d8a859857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9634225Z frame #33: <unknown function> + 0x4a54542 (0x7f0d8a88e542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9634640Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7f0d7ab5590b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.9634864Z frame #35: <unknown function> + 0xdbbf4 (0x7f0daa9bbbf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T03:04:41.9635264Z frame #36: <unknown function> + 0x76db (0x7f0dcb00f6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T03:04:41.9635518Z frame #37: clone + 0x3f (0x7f0dcad3861f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T03:04:41.9635559Z 
2022-11-23T03:04:41.9635578Z 
2022-11-23T03:04:41.9635694Z On WorkerInfo(id=2, name=worker2):
2022-11-23T03:04:41.9647901Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!\nException raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fcb42c8559b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fcb42c80dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7fcb4eecc13d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fcb4eecd67f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fcb4eeceef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fcb4f1c3b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x2a0be3e (0x7fcb45b3be3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #7: <unknown function> + 0x2a0bf46 (0x7fcb45b3bf46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)\nframe #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fcb4fcd5c58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #9: <unknown function> + 0x35f0340 (0x7fcb51548340 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #10: <unknown function> + 0x35f0ab9 (0x7fcb51548ab9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fcb4fd0fe62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #12: <unknown function> + 0x2ff562 (0x7fcb5ac36562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #13: <unknown function> + 0x2ff956 (0x7fcb5ac36956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #14: <unknown function> + 0x1ddc68 (0x5610ec833c68 in /opt/conda/bin/python)\nframe #15: <unknown function> + 0x199499 (0x5610ec7ef499 in /opt/conda/bin/python)\nframe #16: <unknown function> + 0x1995fa (0x5610ec7ef5fa in /opt/conda/bin/python)\nframe #17: PyNumber_Add + 0x41 (0x5610ec79b4b1 in /opt/conda/bin/python)\nframe #18: _PyEval_EvalFrameDefault + 0x1008 (0x5610ec838098 in /opt/conda/bin/python)\nframe #19: <unknown function> + 0x18f742 (0x5610ec7e5742 in /opt/conda/bin/python)\nframe #20: _PyObject_Call + 0x20a (0x5610ec79dfaa in /opt/conda/bin/python)\nframe #21: _PyEval_EvalFrameDefault + 0x26e4 (0x5610ec839774 in /opt/conda/bin/python)\nframe #22: <unknown function> + 0x18f742 (0x5610ec7e5742 in /opt/conda/bin/python)\nframe #23: _PyObject_Call + 0x20a (0x5610ec79dfaa in /opt/conda/bin/python)\nframe #24: <unknown function> + 0xaa8dba (0x7fcb5b3dfdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fcb5b3ddffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fcb5b3e12d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fcb5b3e2b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fcb5298353c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fcb5b3e10c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)\nframe #30: <unknown function> + 0x4a249c3 (0x7fcb5297c9c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fcb5297d558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fcb52977857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #33: <unknown function> + 0x4a54542 (0x7fcb529ac542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)\nframe #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fcb42c7390b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)\nframe #35: <unknown function> + 0xdbbf4 (0x7fcb72ad9bf4 in /opt/conda/bin/../lib/libstdc++.so.6)\nframe #36: <unknown function> + 0x76db (0x7fcb9312d6db in /lib/x86_64-linux-gnu/libpthread.so.0)\nframe #37: clone + 0x3f (0x7fcb92e5661f in /lib/x86_64-linux-gnu/libc.so.6)\n')
2022-11-23T03:04:41.9648163Z Traceback (most recent call last):
2022-11-23T03:04:41.9648525Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2022-11-23T03:04:41.9648716Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2022-11-23T03:04:41.9649118Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2022-11-23T03:04:41.9649238Z     return x.cpu() + y.cuda()
2022-11-23T03:04:41.9649475Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2022-11-23T03:04:41.9649759Z Exception raised from compute_types at /var/lib/jenkins/workspace/aten/src/ATen/TensorIterator.cpp:484 (most recent call first):
2022-11-23T03:04:41.9650318Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fcb42c8559b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.9650937Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7fcb42c80dfe in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.9651436Z frame #2: at::TensorIteratorBase::compute_types(at::TensorIteratorConfig const&) + 0xced (0x7fcb4eecc13d in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9651967Z frame #3: at::TensorIteratorBase::build(at::TensorIteratorConfig&) + 0x7f (0x7fcb4eecd67f in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9652545Z frame #4: at::TensorIteratorBase::build_borrowing_binary_op(at::TensorBase const&, at::TensorBase const&, at::TensorBase const&) + 0xf2 (0x7fcb4eeceef2 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9653062Z frame #5: at::meta::structured_add_Tensor::meta(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x2e (0x7fcb4f1c3b7e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9653446Z frame #6: <unknown function> + 0x2a0be3e (0x7fcb45b3be3e in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.9653828Z frame #7: <unknown function> + 0x2a0bf46 (0x7fcb45b3bf46 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
2022-11-23T03:04:41.9654379Z frame #8: at::_ops::add_Tensor::redispatch(c10::DispatchKeySet, at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x98 (0x7fcb4fcd5c58 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9654759Z frame #9: <unknown function> + 0x35f0340 (0x7fcb51548340 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9655140Z frame #10: <unknown function> + 0x35f0ab9 (0x7fcb51548ab9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9655620Z frame #11: at::_ops::add_Tensor::call(at::Tensor const&, at::Tensor const&, c10::Scalar const&) + 0x172 (0x7fcb4fd0fe62 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9656005Z frame #12: <unknown function> + 0x2ff562 (0x7fcb5ac36562 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9656385Z frame #13: <unknown function> + 0x2ff956 (0x7fcb5ac36956 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9656583Z frame #14: <unknown function> + 0x1ddc68 (0x5610ec833c68 in /opt/conda/bin/python)
2022-11-23T03:04:41.9656770Z frame #15: <unknown function> + 0x199499 (0x5610ec7ef499 in /opt/conda/bin/python)
2022-11-23T03:04:41.9656954Z frame #16: <unknown function> + 0x1995fa (0x5610ec7ef5fa in /opt/conda/bin/python)
2022-11-23T03:04:41.9657197Z frame #17: PyNumber_Add + 0x41 (0x5610ec79b4b1 in /opt/conda/bin/python)
2022-11-23T03:04:41.9657410Z frame #18: _PyEval_EvalFrameDefault + 0x1008 (0x5610ec838098 in /opt/conda/bin/python)
2022-11-23T03:04:41.9657599Z frame #19: <unknown function> + 0x18f742 (0x5610ec7e5742 in /opt/conda/bin/python)
2022-11-23T03:04:41.9657767Z frame #20: _PyObject_Call + 0x20a (0x5610ec79dfaa in /opt/conda/bin/python)
2022-11-23T03:04:41.9657971Z frame #21: _PyEval_EvalFrameDefault + 0x26e4 (0x5610ec839774 in /opt/conda/bin/python)
2022-11-23T03:04:41.9658154Z frame #22: <unknown function> + 0x18f742 (0x5610ec7e5742 in /opt/conda/bin/python)
2022-11-23T03:04:41.9658345Z frame #23: _PyObject_Call + 0x20a (0x5610ec79dfaa in /opt/conda/bin/python)
2022-11-23T03:04:41.9658738Z frame #24: <unknown function> + 0xaa8dba (0x7fcb5b3dfdba in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9659263Z frame #25: torch::distributed::rpc::PythonRpcHandler::runPythonUdf(pybind11::object const&) + 0x7d (0x7fcb5b3ddffd in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9659919Z frame #26: torch::distributed::rpc::RequestCallbackImpl::runPythonFunction(pybind11::object const&, std::vector<c10::Stream, std::allocator<c10::Stream> >, bool) const + 0x85 (0x7fcb5b3e12d5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9660597Z frame #27: torch::distributed::rpc::RequestCallbackImpl::processPythonCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x96 (0x7fcb5b3e2b16 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9661418Z frame #28: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x10c (0x7fcb5298353c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9662202Z frame #29: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fcb5b3e10c5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
2022-11-23T03:04:41.9662582Z frame #30: <unknown function> + 0x4a249c3 (0x7fcb5297c9c3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9663242Z frame #31: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x538 (0x7fcb5297d558 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9663873Z frame #32: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fcb52977857 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9664254Z frame #33: <unknown function> + 0x4a54542 (0x7fcb529ac542 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so)
2022-11-23T03:04:41.9664658Z frame #34: c10::ThreadPool::main_loop(unsigned long) + 0x2db (0x7fcb42c7390b in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so)
2022-11-23T03:04:41.9664881Z frame #35: <unknown function> + 0xdbbf4 (0x7fcb72ad9bf4 in /opt/conda/bin/../lib/libstdc++.so.6)
2022-11-23T03:04:41.9665203Z frame #36: <unknown function> + 0x76db (0x7fcb9312d6db in /lib/x86_64-linux-gnu/libpthread.so.0)
2022-11-23T03:04:41.9665479Z frame #37: clone + 0x3f (0x7fcb92e5661f in /lib/x86_64-linux-gnu/libc.so.6)
2022-11-23T03:04:41.9665500Z 
2022-11-23T03:04:41.9665518Z 
2022-11-23T03:04:41.9665617Z ok (7.914s)
2022-11-23T03:04:41.9665636Z 
2022-11-23T03:04:41.9665898Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9666009Z Ran 1 test in 7.914s
2022-11-23T03:04:41.9666075Z 
2022-11-23T03:04:41.9666171Z OK
2022-11-23T03:04:41.9666190Z 
2022-11-23T03:04:41.9666316Z Generating XML reports...
2022-11-23T03:04:41.9667017Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025822.xml
2022-11-23T03:04:41.9667408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9667588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9667963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9668159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9668413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawm_laqj
2022-11-23T03:04:41.9668685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawm_laqj/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9668705Z 
2022-11-23T03:04:41.9668812Z Running tests...
2022-11-23T03:04:41.9669074Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9669416Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9669721Z   test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9669940Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97644
2022-11-23T03:04:41.9670246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97645
2022-11-23T03:04:41.9670458Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97646
2022-11-23T03:04:41.9670667Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97647
2022-11-23T03:04:41.9671050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9671225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9671588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9671779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9672144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9672315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9672689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9672878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9673237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9673411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9673778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9673946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9674312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9674480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9674844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9675032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9675286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhusszkq
2022-11-23T03:04:41.9675619Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhusszkq/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9675882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabj5tr1n
2022-11-23T03:04:41.9676136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabj5tr1n/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9676391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hyopdnh
2022-11-23T03:04:41.9676658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hyopdnh/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9676909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ke3t2hu
2022-11-23T03:04:41.9677173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ke3t2hu/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9677402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9677632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9677861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9678081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9678213Z fi_getinfo: -61
2022-11-23T03:04:41.9678347Z fi_getinfo: -61
2022-11-23T03:04:41.9678481Z fi_getinfo: -61
2022-11-23T03:04:41.9678613Z fi_getinfo: -61
2022-11-23T03:04:41.9678712Z ok (4.635s)
2022-11-23T03:04:41.9678731Z 
2022-11-23T03:04:41.9678993Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9679158Z Ran 1 test in 4.635s
2022-11-23T03:04:41.9679177Z 
2022-11-23T03:04:41.9679251Z OK
2022-11-23T03:04:41.9679269Z 
2022-11-23T03:04:41.9679392Z Generating XML reports...
2022-11-23T03:04:41.9679942Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025833.xml
2022-11-23T03:04:41.9680316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9680491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9680870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9681059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9681311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8s6z64v
2022-11-23T03:04:41.9681577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8s6z64v/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9681602Z 
2022-11-23T03:04:41.9681691Z Running tests...
2022-11-23T03:04:41.9681954Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9682312Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9682630Z   test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9682850Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97831
2022-11-23T03:04:41.9683064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97832
2022-11-23T03:04:41.9683277Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97833
2022-11-23T03:04:41.9683485Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97834
2022-11-23T03:04:41.9683843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9684019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9684398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9684638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9685009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9685179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9685551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9685742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9686106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9686267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9686642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9686831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9687191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9687362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9687729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9687913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9688168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuwj2qr9y
2022-11-23T03:04:41.9688480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuwj2qr9y/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9688737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphmh178wa
2022-11-23T03:04:41.9688988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9v6vdnm
2022-11-23T03:04:41.9689257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphmh178wa/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9689518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9v6vdnm/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9689769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5v7s8wh1
2022-11-23T03:04:41.9690033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5v7s8wh1/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9690259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9690490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9690699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9690923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9691077Z fi_getinfo: -61
2022-11-23T03:04:41.9691215Z fi_getinfo: -61
2022-11-23T03:04:41.9691350Z fi_getinfo: -61
2022-11-23T03:04:41.9691482Z fi_getinfo: -61
2022-11-23T03:04:41.9691581Z ok (4.644s)
2022-11-23T03:04:41.9691600Z 
2022-11-23T03:04:41.9691846Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9691957Z Ran 1 test in 4.644s
2022-11-23T03:04:41.9691976Z 
2022-11-23T03:04:41.9692070Z OK
2022-11-23T03:04:41.9692089Z 
2022-11-23T03:04:41.9692210Z Generating XML reports...
2022-11-23T03:04:41.9692758Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025840.xml
2022-11-23T03:04:41.9693132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9693305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9693732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9693930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9694168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xfz4twx
2022-11-23T03:04:41.9694435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xfz4twx/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9694455Z 
2022-11-23T03:04:41.9694560Z Running tests...
2022-11-23T03:04:41.9694826Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9695182Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9695512Z   test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9695730Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98018
2022-11-23T03:04:41.9695949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98019
2022-11-23T03:04:41.9696146Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98020
2022-11-23T03:04:41.9696362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98021
2022-11-23T03:04:41.9696733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9696907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9697286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9697530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9697896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9698067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9698441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9698614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9698969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9699138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9699503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9699690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9700058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9700229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9700596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9700766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9701020Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp194df_91
2022-11-23T03:04:41.9701283Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp194df_91/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9701541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7g984pr
2022-11-23T03:04:41.9701807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7g984pr/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9702062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqycjqpj5
2022-11-23T03:04:41.9702330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqycjqpj5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9702633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohtpez1m
2022-11-23T03:04:41.9702905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohtpez1m/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9703116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9703347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9703570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9703796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9703947Z fi_getinfo: -61
2022-11-23T03:04:41.9704046Z ok (10.525s)
2022-11-23T03:04:41.9704066Z 
2022-11-23T03:04:41.9704329Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9704441Z Ran 1 test in 10.525s
2022-11-23T03:04:41.9704460Z 
2022-11-23T03:04:41.9704534Z OK
2022-11-23T03:04:41.9704552Z 
2022-11-23T03:04:41.9704678Z Generating XML reports...
2022-11-23T03:04:41.9705227Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025847.xml
2022-11-23T03:04:41.9705596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9705770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9706144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9706389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9706642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9i5it1_5
2022-11-23T03:04:41.9707069Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9i5it1_5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9707093Z 
2022-11-23T03:04:41.9707197Z Running tests...
2022-11-23T03:04:41.9707466Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9707823Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9708145Z   test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9708361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98233
2022-11-23T03:04:41.9708575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98234
2022-11-23T03:04:41.9708789Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98235
2022-11-23T03:04:41.9709001Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98236
2022-11-23T03:04:41.9709358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9709536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9709918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9710105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9710464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9710634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9711037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9711231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9711589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9711885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9712272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9712458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9712824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9712995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9713360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9713550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9713807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpse19blb8
2022-11-23T03:04:41.9714060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpse19blb8/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9714320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy6lmi3r9
2022-11-23T03:04:41.9714585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy6lmi3r9/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9714835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbsgg9x6s
2022-11-23T03:04:41.9715286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbsgg9x6s/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9715708Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89x9axj2
2022-11-23T03:04:41.9716120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89x9axj2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9716349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9716577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9716790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9717017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9717177Z fi_getinfo: -61
2022-11-23T03:04:41.9717276Z ok (12.168s)
2022-11-23T03:04:41.9717296Z 
2022-11-23T03:04:41.9717559Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9717672Z Ran 1 test in 12.168s
2022-11-23T03:04:41.9717691Z 
2022-11-23T03:04:41.9717781Z OK
2022-11-23T03:04:41.9717800Z 
2022-11-23T03:04:41.9717920Z Generating XML reports...
2022-11-23T03:04:41.9718456Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025900.xml
2022-11-23T03:04:41.9718826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9719003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9719404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9719582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9719831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vbw_07r
2022-11-23T03:04:41.9720101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vbw_07r/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9720121Z 
2022-11-23T03:04:41.9720251Z Running tests...
2022-11-23T03:04:41.9720508Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9720862Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9721189Z   test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9721458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98449
2022-11-23T03:04:41.9721665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98450
2022-11-23T03:04:41.9721875Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98451
2022-11-23T03:04:41.9722083Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98452
2022-11-23T03:04:41.9722459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9722634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9723000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9723192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9723550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9723725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9724104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9724292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9724653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9724826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9725197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9725422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9725787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9725962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9726332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9726516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9726771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8xi6b_i4
2022-11-23T03:04:41.9727038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8xi6b_i4/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9727293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz6in_v52
2022-11-23T03:04:41.9727545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz6in_v52/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9727797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggr4zaf0
2022-11-23T03:04:41.9728065Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggr4zaf0/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9728314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8shn8ndf
2022-11-23T03:04:41.9728580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8shn8ndf/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9728807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9729035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9729262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9729488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9729616Z fi_getinfo: -61
2022-11-23T03:04:41.9729715Z ok (12.462s)
2022-11-23T03:04:41.9729735Z 
2022-11-23T03:04:41.9729996Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9730108Z Ran 1 test in 12.463s
2022-11-23T03:04:41.9730176Z 
2022-11-23T03:04:41.9730271Z OK
2022-11-23T03:04:41.9730290Z 
2022-11-23T03:04:41.9730413Z Generating XML reports...
2022-11-23T03:04:41.9730966Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025915.xml
2022-11-23T03:04:41.9731339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9731497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9731874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9732067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9732322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp05o2fr2x
2022-11-23T03:04:41.9732592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp05o2fr2x/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9732612Z 
2022-11-23T03:04:41.9732718Z Running tests...
2022-11-23T03:04:41.9732980Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9733335Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9733660Z   test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9733863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98665
2022-11-23T03:04:41.9734142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98666
2022-11-23T03:04:41.9734353Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98667
2022-11-23T03:04:41.9734564Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98668
2022-11-23T03:04:41.9734941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9735114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9735495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9735683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9736025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9736195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9736569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9736760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9737118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9737291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9737656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9737844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9738209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9738364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9738744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9738931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9739186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpad3l8jnp
2022-11-23T03:04:41.9739506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpad3l8jnp/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9739766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpucpj2xa1
2022-11-23T03:04:41.9740036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpucpj2xa1/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9740287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbb1f4g_b
2022-11-23T03:04:41.9740535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbb1f4g_b/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9740789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7w9qbrh
2022-11-23T03:04:41.9741052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7w9qbrh/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9741281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9741512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9741737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9741960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9742111Z fi_getinfo: -61
2022-11-23T03:04:41.9742211Z ok (10.547s)
2022-11-23T03:04:41.9742231Z 
2022-11-23T03:04:41.9742476Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9742587Z Ran 1 test in 10.548s
2022-11-23T03:04:41.9742606Z 
2022-11-23T03:04:41.9742752Z OK
2022-11-23T03:04:41.9742771Z 
2022-11-23T03:04:41.9742893Z Generating XML reports...
2022-11-23T03:04:41.9743442Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025931.xml
2022-11-23T03:04:41.9743814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9743990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9744371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9744543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9744798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl2kbmwtk
2022-11-23T03:04:41.9745064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl2kbmwtk/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9745087Z 
2022-11-23T03:04:41.9745198Z Running tests...
2022-11-23T03:04:41.9745459Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9745815Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9746130Z   test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9746348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98880
2022-11-23T03:04:41.9746562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98881
2022-11-23T03:04:41.9746934Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98882
2022-11-23T03:04:41.9747169Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98883
2022-11-23T03:04:41.9747547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9747728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9748103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9748290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9748725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9748905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9749260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9749449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9749807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9749978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9750349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9750532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9750900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9751072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9751437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9751606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9751861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7yimmzd8
2022-11-23T03:04:41.9752130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7yimmzd8/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9752454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxsr9l7dd
2022-11-23T03:04:41.9752722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxsr9l7dd/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9752971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ashfdmr
2022-11-23T03:04:41.9753240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ashfdmr/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9753491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6hszx9t
2022-11-23T03:04:41.9753732Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6hszx9t/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9753960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9754186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9754412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9754636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9754786Z fi_getinfo: -61
2022-11-23T03:04:41.9754921Z fi_getinfo: -61
2022-11-23T03:04:41.9755055Z fi_getinfo: -61
2022-11-23T03:04:41.9755176Z fi_getinfo: -61
2022-11-23T03:04:41.9755277Z ok (17.883s)
2022-11-23T03:04:41.9755296Z 
2022-11-23T03:04:41.9755559Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9755671Z Ran 1 test in 17.883s
2022-11-23T03:04:41.9755690Z 
2022-11-23T03:04:41.9755782Z OK
2022-11-23T03:04:41.9755801Z 
2022-11-23T03:04:41.9755923Z Generating XML reports...
2022-11-23T03:04:41.9756471Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025944.xml
2022-11-23T03:04:41.9756838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9756999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9757379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9757626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9757890Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4323xp_
2022-11-23T03:04:41.9758159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4323xp_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9758178Z 
2022-11-23T03:04:41.9758286Z Running tests...
2022-11-23T03:04:41.9758553Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9758909Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9759223Z   test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9759424Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99227
2022-11-23T03:04:41.9759638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99228
2022-11-23T03:04:41.9759851Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99229
2022-11-23T03:04:41.9760061Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99230
2022-11-23T03:04:41.9760438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9760613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9760991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9761180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9761590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9761766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9762136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9762324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9762685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9762855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9763219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9763406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9763773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9763925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9764289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9764476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9764733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusbxkg8s
2022-11-23T03:04:41.9764986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyk3vjf25
2022-11-23T03:04:41.9765256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusbxkg8s/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9765520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyk3vjf25/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9765772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rnau3nk
2022-11-23T03:04:41.9766023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rnau3nk/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9766274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg133d7dw
2022-11-23T03:04:41.9766583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg133d7dw/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9766817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9767045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9767268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9767492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9767642Z fi_getinfo: -61
2022-11-23T03:04:41.9767762Z fi_getinfo: -61
2022-11-23T03:04:41.9767903Z fi_getinfo: -61
2022-11-23T03:04:41.9768036Z fi_getinfo: -61
2022-11-23T03:04:41.9768134Z ok (20.251s)
2022-11-23T03:04:41.9768153Z 
2022-11-23T03:04:41.9768417Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9768529Z Ran 1 test in 20.251s
2022-11-23T03:04:41.9768548Z 
2022-11-23T03:04:41.9768639Z OK
2022-11-23T03:04:41.9768658Z 
2022-11-23T03:04:41.9768783Z Generating XML reports...
2022-11-23T03:04:41.9769314Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030004.xml
2022-11-23T03:04:41.9769686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9769861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9770239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9770483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9770738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmg7t07_c
2022-11-23T03:04:41.9771007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmg7t07_c/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9771026Z 
2022-11-23T03:04:41.9771136Z Running tests...
2022-11-23T03:04:41.9771402Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9771742Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9772057Z   test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9772818Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81962 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.635s)
2022-11-23T03:04:41.9772843Z 
2022-11-23T03:04:41.9773105Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9773214Z Ran 1 test in 1.635s
2022-11-23T03:04:41.9773233Z 
2022-11-23T03:04:41.9773338Z OK (skipped=1)
2022-11-23T03:04:41.9773360Z 
2022-11-23T03:04:41.9773482Z Generating XML reports...
2022-11-23T03:04:41.9774028Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030027.xml
2022-11-23T03:04:41.9774395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9774569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9774931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9775126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9775380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2xwprsw
2022-11-23T03:04:41.9775647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2xwprsw/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9775667Z 
2022-11-23T03:04:41.9775838Z Running tests...
2022-11-23T03:04:41.9776110Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9776466Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9776777Z   test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9776978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99614
2022-11-23T03:04:41.9777197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99615
2022-11-23T03:04:41.9777414Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99616
2022-11-23T03:04:41.9777623Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99617
2022-11-23T03:04:41.9777992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9778168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9778548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9778735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9779093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9779248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9779616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9779860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9780218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9780393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9780760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9780945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9781311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9781465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9781831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9782020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9782275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9u5sj18
2022-11-23T03:04:41.9782540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9u5sj18/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9782796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgj16zqft
2022-11-23T03:04:41.9783062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgj16zqft/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9783312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgpls_y63
2022-11-23T03:04:41.9783574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgpls_y63/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9783808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw57owlr9
2022-11-23T03:04:41.9784072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw57owlr9/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9784301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9784527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9784807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9785037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9785185Z fi_getinfo: -61
2022-11-23T03:04:41.9785319Z fi_getinfo: -61
2022-11-23T03:04:41.9785436Z fi_getinfo: -61
2022-11-23T03:04:41.9785568Z fi_getinfo: -61
2022-11-23T03:04:41.9785664Z ok (20.365s)
2022-11-23T03:04:41.9785684Z 
2022-11-23T03:04:41.9785940Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9786054Z Ran 1 test in 20.366s
2022-11-23T03:04:41.9786078Z 
2022-11-23T03:04:41.9786176Z OK
2022-11-23T03:04:41.9786195Z 
2022-11-23T03:04:41.9786315Z Generating XML reports...
2022-11-23T03:04:41.9787029Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030032.xml
2022-11-23T03:04:41.9787421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9787598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9787959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9788148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9788396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3r8s1f90
2022-11-23T03:04:41.9788658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3r8s1f90/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9788755Z 
2022-11-23T03:04:41.9788869Z Running tests...
2022-11-23T03:04:41.9789134Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9789490Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9789805Z   test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9790023Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99967
2022-11-23T03:04:41.9790223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99968
2022-11-23T03:04:41.9790437Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99969
2022-11-23T03:04:41.9790644Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99970
2022-11-23T03:04:41.9791018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9791195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9791573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9791766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9792128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9792283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9792653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9792840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9793195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9793368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9793735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9793919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9794347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9794509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9794882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9795069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9795325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5xs5jeq
2022-11-23T03:04:41.9795596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5xs5jeq/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9795852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnd0f4xps
2022-11-23T03:04:41.9796117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnd0f4xps/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9796373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpprb5zih7
2022-11-23T03:04:41.9796636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpprb5zih7/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9796867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrnb96ic
2022-11-23T03:04:41.9797129Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrnb96ic/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9797355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9797580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9797859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9798082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9798231Z fi_getinfo: -61
2022-11-23T03:04:41.9798366Z fi_getinfo: -61
2022-11-23T03:04:41.9798486Z fi_getinfo: -61
2022-11-23T03:04:41.9798620Z fi_getinfo: -61
2022-11-23T03:04:41.9798719Z ok (17.754s)
2022-11-23T03:04:41.9798738Z 
2022-11-23T03:04:41.9798999Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9799109Z Ran 1 test in 17.754s
2022-11-23T03:04:41.9799129Z 
2022-11-23T03:04:41.9799220Z OK
2022-11-23T03:04:41.9799238Z 
2022-11-23T03:04:41.9799360Z Generating XML reports...
2022-11-23T03:04:41.9799903Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030055.xml
2022-11-23T03:04:41.9800260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9800436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9800813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9801005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9801259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8yrv8r1d
2022-11-23T03:04:41.9801524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8yrv8r1d/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9801543Z 
2022-11-23T03:04:41.9801649Z Running tests...
2022-11-23T03:04:41.9801965Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9802306Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9802625Z   test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9802847Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100314
2022-11-23T03:04:41.9803063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100315
2022-11-23T03:04:41.9803327Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100316
2022-11-23T03:04:41.9803548Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100317
2022-11-23T03:04:41.9803925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9804099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9804478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9804655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9805016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9805185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9805554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9805741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9806102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9806275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9806647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9806834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9807240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9807412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9807785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9807970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9808224Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplotwnsws
2022-11-23T03:04:41.9808493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplotwnsws/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9808745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_falowh5
2022-11-23T03:04:41.9809008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_falowh5/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9809247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt2noxhex
2022-11-23T03:04:41.9809513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt2noxhex/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9809759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7utqt14k
2022-11-23T03:04:41.9810023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7utqt14k/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9810249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9810475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9810700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9810923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9811093Z fi_getinfo: -61
2022-11-23T03:04:41.9811239Z fi_getinfo: -61
2022-11-23T03:04:41.9811373Z fi_getinfo: -61
2022-11-23T03:04:41.9811506Z fi_getinfo: -61
2022-11-23T03:04:41.9811605Z ok (16.320s)
2022-11-23T03:04:41.9811624Z 
2022-11-23T03:04:41.9811884Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9811994Z Ran 1 test in 16.320s
2022-11-23T03:04:41.9812013Z 
2022-11-23T03:04:41.9812151Z OK
2022-11-23T03:04:41.9812172Z 
2022-11-23T03:04:41.9812280Z Generating XML reports...
2022-11-23T03:04:41.9812830Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030115.xml
2022-11-23T03:04:41.9813198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9813371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9813746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9813940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9814193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjt4a5t_h
2022-11-23T03:04:41.9814461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjt4a5t_h/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9814480Z 
2022-11-23T03:04:41.9814585Z Running tests...
2022-11-23T03:04:41.9814834Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9815189Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9815501Z   test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9815719Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100660
2022-11-23T03:04:41.9815935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100661
2022-11-23T03:04:41.9816207Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100662
2022-11-23T03:04:41.9816417Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100663
2022-11-23T03:04:41.9816796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9816954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9817330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9817520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9817886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9818055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9818426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9818614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9818972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9819146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9819497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9819684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9820046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9820212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9820574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9820761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9821013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwr6ec6d
2022-11-23T03:04:41.9821334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwr6ec6d/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9821572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp382ad7kx
2022-11-23T03:04:41.9821837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp382ad7kx/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9822088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsi0frbbw
2022-11-23T03:04:41.9822356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsi0frbbw/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9822602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75opfw_d
2022-11-23T03:04:41.9822872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75opfw_d/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9823098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9823328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9823555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9823762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9823908Z fi_getinfo: -61
2022-11-23T03:04:41.9824044Z fi_getinfo: -61
2022-11-23T03:04:41.9824176Z fi_getinfo: -61
2022-11-23T03:04:41.9824310Z fi_getinfo: -61
2022-11-23T03:04:41.9824409Z ok (16.869s)
2022-11-23T03:04:41.9824429Z 
2022-11-23T03:04:41.9824687Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9824837Z Ran 1 test in 16.869s
2022-11-23T03:04:41.9824857Z 
2022-11-23T03:04:41.9824949Z OK
2022-11-23T03:04:41.9824968Z 
2022-11-23T03:04:41.9825090Z Generating XML reports...
2022-11-23T03:04:41.9825640Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030134.xml
2022-11-23T03:04:41.9826012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9826188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9826564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9826753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9827171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4zjn50f
2022-11-23T03:04:41.9827444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4zjn50f/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9827467Z 
2022-11-23T03:04:41.9827576Z Running tests...
2022-11-23T03:04:41.9827848Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9828205Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9828523Z   test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9828744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101009
2022-11-23T03:04:41.9828961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101010
2022-11-23T03:04:41.9829175Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101011
2022-11-23T03:04:41.9829370Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101012
2022-11-23T03:04:41.9829744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9829915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9830293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9830560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9830938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9831113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9831486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9831673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9832014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9832189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9832559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9832748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9833114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9833283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9833648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9833833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9834072Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeu8xlije
2022-11-23T03:04:41.9834416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeu8xlije/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9834670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxyv9s18
2022-11-23T03:04:41.9834937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxyv9s18/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9835191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiw9hzoon
2022-11-23T03:04:41.9835437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vxcm3rv
2022-11-23T03:04:41.9835700Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiw9hzoon/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9835962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vxcm3rv/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9836190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9836406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9836628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9836853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9837001Z fi_getinfo: -61
2022-11-23T03:04:41.9837141Z fi_getinfo: -61
2022-11-23T03:04:41.9837277Z fi_getinfo: -61
2022-11-23T03:04:41.9837411Z fi_getinfo: -61
2022-11-23T03:04:41.9837492Z ok (16.471s)
2022-11-23T03:04:41.9837511Z 
2022-11-23T03:04:41.9837774Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9837885Z Ran 1 test in 16.471s
2022-11-23T03:04:41.9837904Z 
2022-11-23T03:04:41.9837992Z OK
2022-11-23T03:04:41.9838011Z 
2022-11-23T03:04:41.9838133Z Generating XML reports...
2022-11-23T03:04:41.9838679Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030154.xml
2022-11-23T03:04:41.9839052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9839226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9839666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9839848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9840105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8juromj
2022-11-23T03:04:41.9840377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8juromj/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9840397Z 
2022-11-23T03:04:41.9840502Z Running tests...
2022-11-23T03:04:41.9840766Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9841122Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9841441Z   test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9841661Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101358
2022-11-23T03:04:41.9841865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101359
2022-11-23T03:04:41.9842078Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101360
2022-11-23T03:04:41.9842287Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101361
2022-11-23T03:04:41.9842657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9842830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9843205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9843449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9843818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9843992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9844353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9844542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9844900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9845068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9845437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9845626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9845986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9846155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9846508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9846694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9846945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7m96qzti
2022-11-23T03:04:41.9847211Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7m96qzti/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9847465Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0pq2bsm
2022-11-23T03:04:41.9847731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0pq2bsm/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9847987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe40bsqim
2022-11-23T03:04:41.9848252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe40bsqim/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9848550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdk7e47q4
2022-11-23T03:04:41.9848805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdk7e47q4/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9849034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9849259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9849483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9849709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9849863Z fi_getinfo: -61
2022-11-23T03:04:41.9849998Z fi_getinfo: -61
2022-11-23T03:04:41.9850114Z fi_getinfo: -61
2022-11-23T03:04:41.9850247Z fi_getinfo: -61
2022-11-23T03:04:41.9850344Z ok (16.469s)
2022-11-23T03:04:41.9850363Z 
2022-11-23T03:04:41.9850623Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9850737Z Ran 1 test in 16.469s
2022-11-23T03:04:41.9850756Z 
2022-11-23T03:04:41.9850845Z OK
2022-11-23T03:04:41.9850864Z 
2022-11-23T03:04:41.9850984Z Generating XML reports...
2022-11-23T03:04:41.9851530Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030213.xml
2022-11-23T03:04:41.9851901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9852059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9852436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9852738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9852991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1w5gnmcn
2022-11-23T03:04:41.9853261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1w5gnmcn/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9853281Z 
2022-11-23T03:04:41.9853386Z Running tests...
2022-11-23T03:04:41.9853652Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9854006Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9854303Z   test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9854525Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101704
2022-11-23T03:04:41.9854747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101705
2022-11-23T03:04:41.9854960Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101706
2022-11-23T03:04:41.9855169Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101707
2022-11-23T03:04:41.9855543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9855718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9856080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9856250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9856609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9856796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9857173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9857360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9857770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9857949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9858322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9858507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9858851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9859029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9859405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9859591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9859845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_y7myj7l
2022-11-23T03:04:41.9860114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_y7myj7l/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9860367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyj3z091e
2022-11-23T03:04:41.9860617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdg1il1lb
2022-11-23T03:04:41.9860882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyj3z091e/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9861124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdg1il1lb/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9861426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0gomr2ha
2022-11-23T03:04:41.9861688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0gomr2ha/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9861917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9862148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9862374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9862594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9862744Z fi_getinfo: -61
2022-11-23T03:04:41.9862865Z fi_getinfo: -61
2022-11-23T03:04:41.9863000Z fi_getinfo: -61
2022-11-23T03:04:41.9863132Z fi_getinfo: -61
2022-11-23T03:04:41.9863230Z ok (17.885s)
2022-11-23T03:04:41.9863249Z 
2022-11-23T03:04:41.9863512Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9863626Z Ran 1 test in 17.885s
2022-11-23T03:04:41.9863646Z 
2022-11-23T03:04:41.9863735Z OK
2022-11-23T03:04:41.9863754Z 
2022-11-23T03:04:41.9863858Z Generating XML reports...
2022-11-23T03:04:41.9864408Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030232.xml
2022-11-23T03:04:41.9864775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9864948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9865323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9865511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9865761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvhfkgl5y
2022-11-23T03:04:41.9866034Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvhfkgl5y/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9866054Z 
2022-11-23T03:04:41.9866157Z Running tests...
2022-11-23T03:04:41.9866403Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9866991Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9867347Z   test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9867565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102051
2022-11-23T03:04:41.9867782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102052
2022-11-23T03:04:41.9867996Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102053
2022-11-23T03:04:41.9868208Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102054
2022-11-23T03:04:41.9868593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9868767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9869120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9869294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9869670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9869858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9870232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9870419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9870860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9871035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9871388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9871579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9871945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9872116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9872482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9872665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9872920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hdmlbnj
2022-11-23T03:04:41.9873194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hdmlbnj/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9873446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpljke6bgp
2022-11-23T03:04:41.9873698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpljke6bgp/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9873952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppbr8p7mv
2022-11-23T03:04:41.9874214Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppbr8p7mv/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9874461Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9n6cw6e2
2022-11-23T03:04:41.9874720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9n6cw6e2/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9874945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9875176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9875400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9875607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9875800Z fi_getinfo: -61
2022-11-23T03:04:41.9875946Z fi_getinfo: -61
2022-11-23T03:04:41.9876080Z fi_getinfo: -61
2022-11-23T03:04:41.9876213Z fi_getinfo: -61
2022-11-23T03:04:41.9876312Z ok (20.363s)
2022-11-23T03:04:41.9876331Z 
2022-11-23T03:04:41.9876591Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9876686Z Ran 1 test in 20.363s
2022-11-23T03:04:41.9876723Z 
2022-11-23T03:04:41.9876796Z OK
2022-11-23T03:04:41.9876814Z 
2022-11-23T03:04:41.9876934Z Generating XML reports...
2022-11-23T03:04:41.9877481Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030253.xml
2022-11-23T03:04:41.9877855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9878031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9878407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9878597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9878849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9w8558u
2022-11-23T03:04:41.9879098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9w8558u/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9879133Z 
2022-11-23T03:04:41.9879223Z Running tests...
2022-11-23T03:04:41.9879482Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9879895Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9880207Z   test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9880427Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102404
2022-11-23T03:04:41.9880649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102405
2022-11-23T03:04:41.9880862Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102406
2022-11-23T03:04:41.9881071Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102407
2022-11-23T03:04:41.9881426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9881601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9881975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9882168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9882528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9882701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9883074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9883262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9883604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9883773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9884136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9884325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9884688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9884856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9885267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9885461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9885717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9ug3a2z
2022-11-23T03:04:41.9885969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9ug3a2z/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9886222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlps0hlz
2022-11-23T03:04:41.9886493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlps0hlz/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9886745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhl5qqj_
2022-11-23T03:04:41.9887011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhl5qqj_/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9887261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81aaryzu
2022-11-23T03:04:41.9887522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81aaryzu/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9887749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9887959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9888177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9888399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9888600Z fi_getinfo: -61
2022-11-23T03:04:41.9888735Z fi_getinfo: -61
2022-11-23T03:04:41.9888868Z fi_getinfo: -61
2022-11-23T03:04:41.9889001Z fi_getinfo: -61
2022-11-23T03:04:41.9889097Z ok (17.849s)
2022-11-23T03:04:41.9889117Z 
2022-11-23T03:04:41.9889367Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9889480Z Ran 1 test in 17.849s
2022-11-23T03:04:41.9889500Z 
2022-11-23T03:04:41.9889589Z OK
2022-11-23T03:04:41.9889607Z 
2022-11-23T03:04:41.9889727Z Generating XML reports...
2022-11-23T03:04:41.9890270Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030316.xml
2022-11-23T03:04:41.9890638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9890809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9891188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9891362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9891617Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vtpiioc
2022-11-23T03:04:41.9891889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vtpiioc/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9891909Z 
2022-11-23T03:04:41.9892017Z Running tests...
2022-11-23T03:04:41.9892278Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9892632Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9892943Z   test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9893160Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102751
2022-11-23T03:04:41.9893378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102752
2022-11-23T03:04:41.9893577Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102753
2022-11-23T03:04:41.9893789Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102754
2022-11-23T03:04:41.9894206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9894387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9894770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9894958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9895316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9895492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9895852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9896037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9896403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9896573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9896940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9897121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9897477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9897701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9898070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9898236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9898493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpob9bydtv
2022-11-23T03:04:41.9898763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpob9bydtv/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9899016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp233r0kju
2022-11-23T03:04:41.9899280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp233r0kju/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9899532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6gsiz62
2022-11-23T03:04:41.9899794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6gsiz62/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9900046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptcl8njm8
2022-11-23T03:04:41.9900289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptcl8njm8/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9900519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9900744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9900968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9901191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9901337Z fi_getinfo: -61
2022-11-23T03:04:41.9901469Z fi_getinfo: -61
2022-11-23T03:04:41.9901602Z fi_getinfo: -61
2022-11-23T03:04:41.9901719Z fi_getinfo: -61
2022-11-23T03:04:41.9901815Z ok (20.386s)
2022-11-23T03:04:41.9901837Z 
2022-11-23T03:04:41.9902100Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9902210Z Ran 1 test in 20.387s
2022-11-23T03:04:41.9902230Z 
2022-11-23T03:04:41.9902321Z OK
2022-11-23T03:04:41.9902340Z 
2022-11-23T03:04:41.9902461Z Generating XML reports...
2022-11-23T03:04:41.9903067Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030336.xml
2022-11-23T03:04:41.9903452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9903610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9903988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9904177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9904432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmposnf475j
2022-11-23T03:04:41.9904703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmposnf475j/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9904723Z 
2022-11-23T03:04:41.9904828Z Running tests...
2022-11-23T03:04:41.9905088Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9905444Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9905763Z   test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9905966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103104
2022-11-23T03:04:41.9906181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103105
2022-11-23T03:04:41.9906396Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103106
2022-11-23T03:04:41.9906664Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103107
2022-11-23T03:04:41.9907221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9907394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9907784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9907972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9908318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9908490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9908862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9909051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9909412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9909581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9909950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9910137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9910502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9910655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9911057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9911248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9911506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm4cti11e
2022-11-23T03:04:41.9911773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm4cti11e/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9912028Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9z5un3dm
2022-11-23T03:04:41.9912377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9z5un3dm/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9912642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdifi2icz
2022-11-23T03:04:41.9912908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdifi2icz/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9913141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabl5_zt7
2022-11-23T03:04:41.9913400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabl5_zt7/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9913627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9913854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9914076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9914304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9914454Z fi_getinfo: -61
2022-11-23T03:04:41.9914590Z fi_getinfo: -61
2022-11-23T03:04:41.9914709Z fi_getinfo: -61
2022-11-23T03:04:41.9914843Z fi_getinfo: -61
2022-11-23T03:04:41.9914941Z ok (7.834s)
2022-11-23T03:04:41.9914960Z 
2022-11-23T03:04:41.9915222Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9915332Z Ran 1 test in 7.834s
2022-11-23T03:04:41.9915351Z 
2022-11-23T03:04:41.9915441Z OK
2022-11-23T03:04:41.9915459Z 
2022-11-23T03:04:41.9915580Z Generating XML reports...
2022-11-23T03:04:41.9916187Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030359.xml
2022-11-23T03:04:41.9916560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9916737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9917120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9917311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9917564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppw8mgeyx
2022-11-23T03:04:41.9917832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppw8mgeyx/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9917852Z 
2022-11-23T03:04:41.9917956Z Running tests...
2022-11-23T03:04:41.9918218Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9918565Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9918871Z   test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9919094Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103455
2022-11-23T03:04:41.9919310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103456
2022-11-23T03:04:41.9919526Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103457
2022-11-23T03:04:41.9919736Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103458
2022-11-23T03:04:41.9920108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9920281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9920647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9920838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9921203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9921423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9921807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9921997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9922356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9922526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9922890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9923064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9923426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9923598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9923964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9924149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9924403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58w61ufd
2022-11-23T03:04:41.9924669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58w61ufd/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9924922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcis9yiw
2022-11-23T03:04:41.9925245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcis9yiw/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9925479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ky6fbbe
2022-11-23T03:04:41.9925747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ky6fbbe/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9925998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhiyfk95
2022-11-23T03:04:41.9926266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhiyfk95/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9926493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9926717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9926942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9927169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9927301Z fi_getinfo: -61
2022-11-23T03:04:41.9927436Z fi_getinfo: -61
2022-11-23T03:04:41.9927571Z fi_getinfo: -61
2022-11-23T03:04:41.9927704Z fi_getinfo: -61
2022-11-23T03:04:41.9927803Z ok (9.522s)
2022-11-23T03:04:41.9927822Z 
2022-11-23T03:04:41.9928088Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9928200Z Ran 1 test in 9.522s
2022-11-23T03:04:41.9928220Z 
2022-11-23T03:04:41.9928293Z OK
2022-11-23T03:04:41.9928328Z 
2022-11-23T03:04:41.9928433Z Generating XML reports...
2022-11-23T03:04:41.9928973Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030410.xml
2022-11-23T03:04:41.9929338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9929516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9929893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9930081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9930385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpww61wqr7
2022-11-23T03:04:41.9930658Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpww61wqr7/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9930678Z 
2022-11-23T03:04:41.9930768Z Running tests...
2022-11-23T03:04:41.9931031Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9931386Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9931706Z   test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9931930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104102
2022-11-23T03:04:41.9932147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104103
2022-11-23T03:04:41.9932365Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104104
2022-11-23T03:04:41.9932580Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104105
2022-11-23T03:04:41.9932937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9933111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9933485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9933675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9934038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9934262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9934635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9934828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9935191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9935344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9935709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9935892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9936255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9936428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9936791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9936975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9937231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqhmt3uw3
2022-11-23T03:04:41.9937499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqhmt3uw3/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9937738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb4cagahj
2022-11-23T03:04:41.9938007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb4cagahj/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9938259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54xhh7e3
2022-11-23T03:04:41.9938527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54xhh7e3/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9938776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78bo5hwz
2022-11-23T03:04:41.9939035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78bo5hwz/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9939306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9939541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9939750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9939975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9940123Z skip: Need at least 4 CUDA devices (4.349s)
2022-11-23T03:04:41.9940143Z 
2022-11-23T03:04:41.9940409Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9940524Z Ran 1 test in 4.349s
2022-11-23T03:04:41.9940543Z 
2022-11-23T03:04:41.9940649Z OK (skipped=1)
2022-11-23T03:04:41.9940668Z 
2022-11-23T03:04:41.9940790Z Generating XML reports...
2022-11-23T03:04:41.9941352Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123030422.xml
2022-11-23T03:04:41.9941722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9941880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9942258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9942446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9942696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8axthve
2022-11-23T03:04:41.9943018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8axthve/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9943037Z 
2022-11-23T03:04:41.9943145Z Running tests...
2022-11-23T03:04:41.9943408Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9943768Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9944076Z   test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9944297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104273
2022-11-23T03:04:41.9944514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104274
2022-11-23T03:04:41.9944725Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104275
2022-11-23T03:04:41.9944933Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104276
2022-11-23T03:04:41.9945308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9945481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9945854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9946050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9946394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9946563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9947171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9947376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9947752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9947917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9948285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9948546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9948927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9949080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9949450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9949635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9949894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp1dpp09w
2022-11-23T03:04:41.9950171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp1dpp09w/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9950423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9ca3foy
2022-11-23T03:04:41.9950692Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9ca3foy/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9950941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0jxjpgw
2022-11-23T03:04:41.9951189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0jxjpgw/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9951434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkg3eq1g
2022-11-23T03:04:41.9951694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkg3eq1g/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9951921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9952214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9952440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9952666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9952817Z skip: Need at least 4 CUDA devices (4.458s)
2022-11-23T03:04:41.9952838Z 
2022-11-23T03:04:41.9953108Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9953203Z Ran 1 test in 4.458s
2022-11-23T03:04:41.9953222Z 
2022-11-23T03:04:41.9953329Z OK (skipped=1)
2022-11-23T03:04:41.9953348Z 
2022-11-23T03:04:41.9953470Z Generating XML reports...
2022-11-23T03:04:41.9954037Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123030429.xml
2022-11-23T03:04:41.9954405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9954623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9955318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9955523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9955763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mlkbi8j
2022-11-23T03:04:41.9956031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mlkbi8j/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9956051Z 
2022-11-23T03:04:41.9956156Z Running tests...
2022-11-23T03:04:41.9956430Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9956787Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2022-11-23T03:04:41.9957120Z   test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2022-11-23T03:04:41.9957339Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104444
2022-11-23T03:04:41.9957555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104445
2022-11-23T03:04:41.9957834Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104446
2022-11-23T03:04:41.9958038Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104447
2022-11-23T03:04:41.9958435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9958591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9958985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9959161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9959526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9959696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9960063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9960219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9960596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9960784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9961156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9961344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9961777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:121: UserWarning: loaded 51 slow tests
2022-11-23T03:04:41.9961949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-11-23T03:04:41.9962315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:125: UserWarning: loaded 420 disabled tests
2022-11-23T03:04:41.9962505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-11-23T03:04:41.9962745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl82ggqv1
2022-11-23T03:04:41.9963013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl82ggqv1/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9963263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbdi_4fi
2022-11-23T03:04:41.9963527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbdi_4fi/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9963776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht_yu7ke
2022-11-23T03:04:41.9964041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht_yu7ke/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9964290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6tgkls16
2022-11-23T03:04:41.9964556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6tgkls16/_remote_module_non_scriptable.py
2022-11-23T03:04:41.9964784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2022-11-23T03:04:41.9964993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-11-23T03:04:41.9965217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-11-23T03:04:41.9965435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2022-11-23T03:04:41.9965582Z skip: Need at least 4 CUDA devices (4.463s)
2022-11-23T03:04:41.9965604Z 
2022-11-23T03:04:41.9965872Z ----------------------------------------------------------------------
2022-11-23T03:04:41.9965983Z Ran 1 test in 4.463s
2022-11-23T03:04:41.9966002Z 
2022-11-23T03:04:41.9966108Z OK (skipped=1)
2022-11-23T03:04:41.9966127Z 
2022-11-23T03:04:41.9966250Z Generating XML reports...
2022-11-23T03:04:41.9966861Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123030436.xml
2022-11-23T03:04:41.9966901Z 
2022-11-23T03:04:41.9967383Z ##[endgroup]
2022-11-23T03:04:41.9967904Z FINISHED PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_gdovk1ur)
2022-11-23T03:04:41.9967924Z 
2022-11-23T03:04:42.1228318Z 
2022-11-23T03:04:42.1228671Z real	103m16.783s
2022-11-23T03:04:42.1228981Z user	177m19.681s
2022-11-23T03:04:42.1229216Z sys	93m56.576s
2022-11-23T03:04:42.1229465Z + assert_git_not_dirty
2022-11-23T03:04:42.1230006Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]]
2022-11-23T03:04:42.1230428Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *xla* ]]
2022-11-23T03:04:42.1232158Z ++ git status --porcelain
2022-11-23T03:04:43.5385554Z + git_status=
2022-11-23T03:04:43.5386254Z + [[ -n '' ]]
2022-11-23T03:04:43.5387408Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-11-23T03:04:43.5387834Z + [[ 1 == 1 ]]
2022-11-23T03:04:43.5388164Z + echo 'Testing distributed C++ tests'
2022-11-23T03:04:43.5388460Z Testing distributed C++ tests
2022-11-23T03:04:43.5389839Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T03:04:43.5401040Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T03:04:43.5410988Z + TEST_REPORTS_DIR=test/test-reports/cpp-distributed/test_distributed
2022-11-23T03:04:43.5411761Z + mkdir -p test/test-reports/cpp-distributed/test_distributed
2022-11-23T03:04:43.5425712Z + /opt/conda/lib/python3.10/site-packages/torch/bin/FileStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml
2022-11-23T03:04:43.8748941Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-11-23T03:04:43.8749789Z [==========] Running 4 tests from 1 test suite.
2022-11-23T03:04:43.8750222Z [----------] Global test environment set-up.
2022-11-23T03:04:43.8750647Z [----------] 4 tests from FileStoreTest
2022-11-23T03:04:43.8751025Z [ RUN      ] FileStoreTest.testGetAndSet
2022-11-23T03:04:43.8754732Z [       OK ] FileStoreTest.testGetAndSet (0 ms)
2022-11-23T03:04:43.8755209Z [ RUN      ] FileStoreTest.testGetAndSetWithPrefix
2022-11-23T03:04:43.8759836Z [       OK ] FileStoreTest.testGetAndSetWithPrefix (0 ms)
2022-11-23T03:04:43.8760320Z [ RUN      ] FileStoreTest.testStressStore
2022-11-23T03:04:43.8962174Z [       OK ] FileStoreTest.testStressStore (20 ms)
2022-11-23T03:04:43.8962657Z [ RUN      ] FileStoreTest.testStressStoreWithPrefix
2022-11-23T03:04:43.9162988Z [       OK ] FileStoreTest.testStressStoreWithPrefix (20 ms)
2022-11-23T03:04:43.9163820Z [----------] 4 tests from FileStoreTest (41 ms total)
2022-11-23T03:04:43.9164049Z 
2022-11-23T03:04:43.9164281Z [----------] Global test environment tear-down
2022-11-23T03:04:43.9164721Z [==========] 4 tests from 1 test suite ran. (41 ms total)
2022-11-23T03:04:43.9165055Z [  PASSED  ] 4 tests.
2022-11-23T03:04:43.9848053Z + /opt/conda/lib/python3.10/site-packages/torch/bin/HashStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml
2022-11-23T03:04:44.3174836Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-11-23T03:04:44.3175885Z [==========] Running 4 tests from 1 test suite.
2022-11-23T03:04:44.3176297Z [----------] Global test environment set-up.
2022-11-23T03:04:44.3176692Z [----------] 4 tests from HashStoreTest
2022-11-23T03:04:44.3177086Z [ RUN      ] HashStoreTest.testGetAndSet
2022-11-23T03:04:44.4180408Z [       OK ] HashStoreTest.testGetAndSet (100 ms)
2022-11-23T03:04:44.4180915Z [ RUN      ] HashStoreTest.testGetAndSetWithPrefix
2022-11-23T03:04:44.5183538Z [       OK ] HashStoreTest.testGetAndSetWithPrefix (100 ms)
2022-11-23T03:04:44.5184021Z [ RUN      ] HashStoreTest.testStressStore
2022-11-23T03:04:44.5191302Z [       OK ] HashStoreTest.testStressStore (0 ms)
2022-11-23T03:04:44.5192149Z [ RUN      ] HashStoreTest.testStressStoreWithPrefix
2022-11-23T03:04:44.5198654Z [       OK ] HashStoreTest.testStressStoreWithPrefix (0 ms)
2022-11-23T03:04:44.5199410Z [----------] 4 tests from HashStoreTest (202 ms total)
2022-11-23T03:04:44.5199641Z 
2022-11-23T03:04:44.5199858Z [----------] Global test environment tear-down
2022-11-23T03:04:44.5201073Z [==========] 4 tests from 1 test suite ran. (202 ms total)
2022-11-23T03:04:44.5201442Z [  PASSED  ] 4 tests.
2022-11-23T03:04:44.5883708Z + /opt/conda/lib/python3.10/site-packages/torch/bin/TCPStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml
2022-11-23T03:04:44.9150301Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-11-23T03:04:44.9151235Z [==========] Running 11 tests from 1 test suite.
2022-11-23T03:04:44.9151666Z [----------] Global test environment set-up.
2022-11-23T03:04:44.9152066Z [----------] 11 tests from TCPStoreTest
2022-11-23T03:04:44.9152466Z [ RUN      ] TCPStoreTest.testHelper
2022-11-23T03:04:45.8631126Z [       OK ] TCPStoreTest.testHelper (947 ms)
2022-11-23T03:04:45.8631589Z [ RUN      ] TCPStoreTest.testHelperPrefix
2022-11-23T03:04:46.8171725Z [       OK ] TCPStoreTest.testHelperPrefix (954 ms)
2022-11-23T03:04:46.8172215Z [ RUN      ] TCPStoreTest.testWatchKeyCallback
2022-11-23T03:04:46.8315904Z [       OK ] TCPStoreTest.testWatchKeyCallback (14 ms)
2022-11-23T03:04:46.8316420Z [ RUN      ] TCPStoreTest.testWatchKeyCallbackWithPrefix
2022-11-23T03:04:46.8460087Z [       OK ] TCPStoreTest.testWatchKeyCallbackWithPrefix (14 ms)
2022-11-23T03:04:46.8460617Z [ RUN      ] TCPStoreTest.testKeyEmptyUpdate
2022-11-23T03:04:47.0516246Z [       OK ] TCPStoreTest.testKeyEmptyUpdate (205 ms)
2022-11-23T03:04:47.0516735Z [ RUN      ] TCPStoreTest.testKeyUpdate
2022-11-23T03:04:47.0522604Z [       OK ] TCPStoreTest.testKeyUpdate (0 ms)
2022-11-23T03:04:47.0523034Z [ RUN      ] TCPStoreTest.testKeyCreate
2022-11-23T03:04:47.0528061Z [       OK ] TCPStoreTest.testKeyCreate (0 ms)
2022-11-23T03:04:47.0528498Z [ RUN      ] TCPStoreTest.testKeyAdd
2022-11-23T03:04:47.0533506Z [       OK ] TCPStoreTest.testKeyAdd (0 ms)
2022-11-23T03:04:47.0533933Z [ RUN      ] TCPStoreTest.testKeyDelete
2022-11-23T03:04:47.2593973Z [       OK ] TCPStoreTest.testKeyDelete (205 ms)
2022-11-23T03:04:47.2594472Z [ RUN      ] TCPStoreTest.testCleanShutdown
2022-11-23T03:04:47.2603020Z [       OK ] TCPStoreTest.testCleanShutdown (0 ms)
2022-11-23T03:04:47.2603539Z [ RUN      ] TCPStoreTest.testMultiTenantStores
2022-11-23T03:04:47.2617902Z [       OK ] TCPStoreTest.testMultiTenantStores (1 ms)
2022-11-23T03:04:47.2618408Z [----------] 11 tests from TCPStoreTest (2346 ms total)
2022-11-23T03:04:47.2618648Z 
2022-11-23T03:04:47.2619061Z [----------] Global test environment tear-down
2022-11-23T03:04:47.2621636Z [==========] 11 tests from 1 test suite ran. (2346 ms total)
2022-11-23T03:04:47.2622041Z [  PASSED  ] 11 tests.
2022-11-23T03:04:47.3367072Z ++ command -v mpiexec
2022-11-23T03:04:47.3369744Z + MPIEXEC=/usr/bin/mpiexec
2022-11-23T03:04:47.3370444Z + [[ -n /usr/bin/mpiexec ]]
2022-11-23T03:04:47.3370753Z + [[ -z true ]]
2022-11-23T03:04:47.3371431Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml
2022-11-23T03:04:47.6651468Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-11-23T03:04:47.6652268Z [==========] Running 12 tests from 1 test suite.
2022-11-23T03:04:47.6652699Z [----------] Global test environment set-up.
2022-11-23T03:04:47.6653129Z [----------] 12 tests from ProcessGroupGlooTest
2022-11-23T03:04:47.6653601Z [ RUN      ] ProcessGroupGlooTest.testSIGSTOPException
2022-11-23T03:04:48.7157717Z [       OK ] ProcessGroupGlooTest.testSIGSTOPException (1050 ms)
2022-11-23T03:04:48.7158296Z [ RUN      ] ProcessGroupGlooTest.testSIGKILLException
2022-11-23T03:04:48.7494799Z [       OK ] ProcessGroupGlooTest.testSIGKILLException (33 ms)
2022-11-23T03:04:48.7495328Z [ RUN      ] ProcessGroupGlooTest.testAllReduceCPU
2022-11-23T03:04:49.0276426Z [       OK ] ProcessGroupGlooTest.testAllReduceCPU (278 ms)
2022-11-23T03:04:49.0276957Z [ RUN      ] ProcessGroupGlooTest.testBroadcastCPU
2022-11-23T03:04:49.0700977Z [       OK ] ProcessGroupGlooTest.testBroadcastCPU (42 ms)
2022-11-23T03:04:49.0701499Z [ RUN      ] ProcessGroupGlooTest.testAllToAllCPU
2022-11-23T03:04:49.2137451Z [       OK ] ProcessGroupGlooTest.testAllToAllCPU (143 ms)
2022-11-23T03:04:49.2137977Z [ RUN      ] ProcessGroupGlooTest.testBarrier
2022-11-23T03:04:49.2554112Z [       OK ] ProcessGroupGlooTest.testBarrier (41 ms)
2022-11-23T03:04:49.2554636Z [ RUN      ] ProcessGroupGlooTest.testMonitoredBarrier
2022-11-23T03:04:50.2758767Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 1000 ms
2022-11-23T03:04:50.2964335Z [       OK ] ProcessGroupGlooTest.testMonitoredBarrier (1040 ms)
2022-11-23T03:04:50.2964920Z [ RUN      ] ProcessGroupGlooTest.testSequenceNumInit
2022-11-23T03:04:50.4391656Z [       OK ] ProcessGroupGlooTest.testSequenceNumInit (142 ms)
2022-11-23T03:04:50.4392173Z [ RUN      ] ProcessGroupGlooTest.testSend
2022-11-23T03:04:50.4810877Z [       OK ] ProcessGroupGlooTest.testSend (41 ms)
2022-11-23T03:04:50.4811343Z [ RUN      ] ProcessGroupGlooTest.testRecv
2022-11-23T03:04:50.5228098Z [       OK ] ProcessGroupGlooTest.testRecv (41 ms)
2022-11-23T03:04:50.5228556Z [ RUN      ] ProcessGroupGlooTest.testStoreSetGet
2022-11-23T03:04:50.5640725Z [       OK ] ProcessGroupGlooTest.testStoreSetGet (41 ms)
2022-11-23T03:04:50.5641230Z [ RUN      ] ProcessGroupGlooTest.testWaitDelay
2022-11-23T03:04:50.7163871Z [       OK ] ProcessGroupGlooTest.testWaitDelay (152 ms)
2022-11-23T03:04:50.7164386Z [----------] 12 tests from ProcessGroupGlooTest (3051 ms total)
2022-11-23T03:04:50.7164637Z 
2022-11-23T03:04:50.7164981Z [----------] Global test environment tear-down
2022-11-23T03:04:50.7167385Z [==========] 12 tests from 1 test suite ran. (3051 ms total)
2022-11-23T03:04:50.7167751Z [  PASSED  ] 12 tests.
2022-11-23T03:04:50.8008825Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml
2022-11-23T03:04:51.2311769Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-11-23T03:04:51.2312635Z [==========] Running 11 tests from 1 test suite.
2022-11-23T03:04:51.2313190Z [----------] Global test environment set-up.
2022-11-23T03:04:51.2313668Z [----------] 11 tests from ProcessGroupNCCLTest
2022-11-23T03:04:51.2314153Z [ RUN      ] ProcessGroupNCCLTest.testAllreduce
2022-11-23T03:04:55.7449193Z [       OK ] ProcessGroupNCCLTest.testAllreduce (4513 ms)
2022-11-23T03:04:55.7449736Z [ RUN      ] ProcessGroupNCCLTest.testBroadcast
2022-11-23T03:04:59.2545643Z [       OK ] ProcessGroupNCCLTest.testBroadcast (3509 ms)
2022-11-23T03:04:59.2546165Z [ RUN      ] ProcessGroupNCCLTest.testReduce
2022-11-23T03:05:02.7643907Z [       OK ] ProcessGroupNCCLTest.testReduce (3509 ms)
2022-11-23T03:05:02.7644432Z [ RUN      ] ProcessGroupNCCLTest.testAllgather
2022-11-23T03:05:04.5560051Z [       OK ] ProcessGroupNCCLTest.testAllgather (1791 ms)
2022-11-23T03:05:04.5560594Z [ RUN      ] ProcessGroupNCCLTest.testAllgatherBase
2022-11-23T03:05:06.3385484Z [       OK ] ProcessGroupNCCLTest.testAllgatherBase (1782 ms)
2022-11-23T03:05:06.3386014Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatter
2022-11-23T03:05:08.1226588Z [       OK ] ProcessGroupNCCLTest.testReduceScatter (1784 ms)
2022-11-23T03:05:08.1227564Z [ RUN      ] ProcessGroupNCCLTest.testSequenceNumInit
2022-11-23T03:05:08.2340738Z [       OK ] ProcessGroupNCCLTest.testSequenceNumInit (111 ms)
2022-11-23T03:05:08.2341428Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout
2022-11-23T03:05:11.2384673Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout (3004 ms)
2022-11-23T03:05:11.2385452Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException
2022-11-23T03:05:14.2397614Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException (3001 ms)
2022-11-23T03:05:14.2398294Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatterBase
2022-11-23T03:05:16.4050779Z [       OK ] ProcessGroupNCCLTest.testReduceScatterBase (2165 ms)
2022-11-23T03:05:16.4051347Z [ RUN      ] ProcessGroupNCCLTest.testBackendName
2022-11-23T03:05:16.4486295Z [       OK ] ProcessGroupNCCLTest.testBackendName (43 ms)
2022-11-23T03:05:16.4487061Z [----------] 11 tests from ProcessGroupNCCLTest (25217 ms total)
2022-11-23T03:05:16.4487313Z 
2022-11-23T03:05:16.4487547Z [----------] Global test environment tear-down
2022-11-23T03:05:16.4488308Z [==========] 11 tests from 1 test suite ran. (25217 ms total)
2022-11-23T03:05:16.4488680Z [  PASSED  ] 11 tests.
2022-11-23T03:05:16.6677304Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml
2022-11-23T03:05:17.0943093Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-11-23T03:05:17.0943849Z [==========] Running 3 tests from 1 test suite.
2022-11-23T03:05:17.0944268Z [----------] Global test environment set-up.
2022-11-23T03:05:17.0944708Z [----------] 3 tests from ProcessGroupNCCLErrorsTest
2022-11-23T03:05:17.0945234Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking
2022-11-23T03:05:19.5085041Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking (2414 ms)
2022-11-23T03:05:19.5085729Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking
2022-11-23T03:05:22.6149304Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking (3106 ms)
2022-11-23T03:05:22.6150046Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking
2022-11-23T03:05:22.7260713Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking (111 ms)
2022-11-23T03:05:22.7262186Z [----------] 3 tests from ProcessGroupNCCLErrorsTest (5631 ms total)
2022-11-23T03:05:22.7262483Z 
2022-11-23T03:05:22.7262737Z [----------] Global test environment tear-down
2022-11-23T03:05:22.7263164Z [==========] 3 tests from 1 test suite ran. (5631 ms total)
2022-11-23T03:05:22.7263512Z [  PASSED  ] 3 tests.
2022-11-23T03:05:22.9410189Z + [[ 1 == 1 ]]
2022-11-23T03:05:22.9410706Z + test_rpc
2022-11-23T03:05:22.9411655Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]]
2022-11-23T03:05:22.9412146Z + echo 'Testing RPC C++ tests'
2022-11-23T03:05:22.9412434Z Testing RPC C++ tests
2022-11-23T03:05:22.9414270Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T03:05:22.9427895Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T03:05:22.9440094Z + ln -sf '/opt/conda/lib/python3.10/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.10/site-packages/torch/bin
2022-11-23T03:05:22.9450371Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc
2022-11-23T03:05:22.9451058Z + mkdir -p test/test-reports/cpp-rpc/test_rpc
2022-11-23T03:05:22.9463711Z + /opt/conda/lib/python3.10/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml
2022-11-23T03:05:23.4066393Z [==========] Running 8 tests from 3 test suites.
2022-11-23T03:05:23.4067809Z [----------] Global test environment set-up.
2022-11-23T03:05:23.4068564Z [----------] 4 tests from WireSerialize
2022-11-23T03:05:23.4069218Z [ RUN      ] WireSerialize.Base
2022-11-23T03:05:23.4084013Z [       OK ] WireSerialize.Base (1 ms)
2022-11-23T03:05:23.4084519Z [ RUN      ] WireSerialize.RecopySparseTensors
2022-11-23T03:05:23.4191586Z [       OK ] WireSerialize.RecopySparseTensors (10 ms)
2022-11-23T03:05:23.4192087Z [ RUN      ] WireSerialize.CloneSparseTensors
2022-11-23T03:05:23.4290252Z [       OK ] WireSerialize.CloneSparseTensors (9 ms)
2022-11-23T03:05:23.4290704Z [ RUN      ] WireSerialize.Errors
2022-11-23T03:05:23.4317855Z [       OK ] WireSerialize.Errors (2 ms)
2022-11-23T03:05:23.4318317Z [----------] 4 tests from WireSerialize (25 ms total)
2022-11-23T03:05:23.4318540Z 
2022-11-23T03:05:23.4318761Z [----------] 1 test from TestE2ETensorPipe
2022-11-23T03:05:23.4319201Z [ RUN      ] TestE2ETensorPipe.TestTrainingLoop
2022-11-23T03:05:24.1452033Z [W tensorpipe_agent.cpp:725] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-11-23T03:05:24.1471675Z [       OK ] TestE2ETensorPipe.TestTrainingLoop (715 ms)
2022-11-23T03:05:24.1472713Z [----------] 1 test from TestE2ETensorPipe (715 ms total)
2022-11-23T03:05:24.1473162Z 
2022-11-23T03:05:24.1473442Z [----------] 3 tests from TensorpipeSerialize
2022-11-23T03:05:24.1473857Z [ RUN      ] TensorpipeSerialize.Base
2022-11-23T03:05:24.1474296Z [       OK ] TensorpipeSerialize.Base (0 ms)
2022-11-23T03:05:24.1474738Z [ RUN      ] TensorpipeSerialize.RecopySparseTensors
2022-11-23T03:05:24.1576265Z [       OK ] TensorpipeSerialize.RecopySparseTensors (10 ms)
2022-11-23T03:05:24.1577246Z [ RUN      ] TensorpipeSerialize.NoDeleterTensors
2022-11-23T03:05:24.1578386Z [       OK ] TensorpipeSerialize.NoDeleterTensors (0 ms)
2022-11-23T03:05:24.1579349Z [----------] 3 tests from TensorpipeSerialize (10 ms total)
2022-11-23T03:05:24.1579779Z 
2022-11-23T03:05:24.1580236Z [----------] Global test environment tear-down
2022-11-23T03:05:24.1581035Z [==========] 8 tests from 3 test suites ran. (751 ms total)
2022-11-23T03:05:24.1581641Z [  PASSED  ] 8 tests.
2022-11-23T03:05:24.1581939Z 
2022-11-23T03:05:24.1582329Z   YOU HAVE 1 DISABLED TEST
2022-11-23T03:05:24.1582646Z 
2022-11-23T03:05:24.2636237Z 
2022-11-23T03:05:24.2687262Z Prepare all required actions
2022-11-23T03:05:24.2687681Z Getting action download info
2022-11-23T03:05:24.4331052Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2022-11-23T03:05:24.6436855Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-11-23T03:05:24.6437144Z with:
2022-11-23T03:05:24.6437632Z   github-token: ***
2022-11-23T03:05:24.6437892Z env:
2022-11-23T03:05:24.6438119Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:05:24.6438395Z   GPU_FLAG: --gpus all
2022-11-23T03:05:24.6438757Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:05:24.6439078Z ##[endgroup]
2022-11-23T03:05:24.6474634Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2022-11-23T03:05:24.6474969Z with:
2022-11-23T03:05:24.6475203Z   shell: bash
2022-11-23T03:05:24.6475433Z   timeout_minutes: 10
2022-11-23T03:05:24.6475693Z   max_attempts: 5
2022-11-23T03:05:24.6476167Z   retry_wait_seconds: 30
2022-11-23T03:05:24.6476671Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"

2022-11-23T03:05:24.6477190Z   polling_interval_seconds: 1
2022-11-23T03:05:24.6477472Z   warning_on_retry: true
2022-11-23T03:05:24.6477748Z   continue_on_error: false
2022-11-23T03:05:24.6477977Z env:
2022-11-23T03:05:24.6478228Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:05:24.6478697Z   GPU_FLAG: --gpus all
2022-11-23T03:05:24.6479027Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:05:24.6479515Z   GITHUB_TOKEN: ***
2022-11-23T03:05:24.6479775Z ##[endgroup]
2022-11-23T03:05:24.7188402Z + python3 -m pip install requests==2.26.0
2022-11-23T03:05:25.0093878Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T03:05:25.1474668Z Collecting requests==2.26.0
2022-11-23T03:05:25.1941352Z   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
2022-11-23T03:05:25.2661259Z Collecting idna<4,>=2.5; python_version >= "3"
2022-11-23T03:05:25.2708375Z   Downloading idna-3.4-py3-none-any.whl (61 kB)
2022-11-23T03:05:25.3364658Z Collecting certifi>=2017.4.17
2022-11-23T03:05:25.3491607Z   Downloading certifi-2022.9.24-py3-none-any.whl (161 kB)
2022-11-23T03:05:25.4562812Z Collecting urllib3<1.27,>=1.21.1
2022-11-23T03:05:25.4896378Z   Downloading urllib3-1.26.12-py2.py3-none-any.whl (140 kB)
2022-11-23T03:05:25.6914521Z Collecting charset-normalizer~=2.0.0; python_version >= "3"
2022-11-23T03:05:25.6954875Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2022-11-23T03:05:25.7824353Z Installing collected packages: idna, certifi, urllib3, charset-normalizer, requests
2022-11-23T03:05:25.9902502Z   WARNING: The script normalizer is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:05:25.9903187Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:05:26.0426712Z Successfully installed certifi-2022.9.24 charset-normalizer-2.0.12 idna-3.4 requests-2.26.0 urllib3-1.26.12
2022-11-23T03:05:26.0905418Z ++ python3 .github/scripts/get_workflow_job_id.py 3528293562 i-099ed1720beaf2723
2022-11-23T03:05:28.0159219Z + GHA_WORKFLOW_JOB_ID=9655200056
2022-11-23T03:05:28.0159781Z + echo job-id=9655200056
2022-11-23T03:05:28.7200672Z Command completed after 1 attempt(s).
2022-11-23T03:05:28.7355199Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-11-23T03:05:28.7355584Z kill "$MONITOR_SCRIPT_PID"
2022-11-23T03:05:28.7369677Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:05:28.7369998Z env:
2022-11-23T03:05:28.7370256Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:05:28.7370519Z   GPU_FLAG: --gpus all
2022-11-23T03:05:28.7370880Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:05:28.7371255Z   MONITOR_SCRIPT_PID: 91496
2022-11-23T03:05:28.7371506Z ##[endgroup]
2022-11-23T03:05:28.7480054Z Prepare all required actions
2022-11-23T03:05:28.7480474Z Getting action download info
2022-11-23T03:05:28.9791085Z Download action repository 'actions/upload-artifact@v3' (SHA:83fd05a356d7e2593de66fc9913b3002723633cb)
2022-11-23T03:05:29.1245214Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-11-23T03:05:29.1245520Z with:
2022-11-23T03:05:29.1245879Z   file-suffix: test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655200056
2022-11-23T03:05:29.1246208Z env:
2022-11-23T03:05:29.1246452Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:05:29.1246726Z   GPU_FLAG: --gpus all
2022-11-23T03:05:29.1247054Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:05:29.1247385Z ##[endgroup]
2022-11-23T03:05:29.1279544Z ##[group]Run # Remove any previous test jsons if they exist
2022-11-23T03:05:29.1279933Z # Remove any previous test jsons if they exist
2022-11-23T03:05:29.1280404Z rm -f test-jsons-*.zip
2022-11-23T03:05:29.1280789Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-11-23T03:05:29.1292693Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:05:29.1292999Z env:
2022-11-23T03:05:29.1293249Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:05:29.1293507Z   GPU_FLAG: --gpus all
2022-11-23T03:05:29.1293861Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:05:29.1294327Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655200056
2022-11-23T03:05:29.1294666Z ##[endgroup]
2022-11-23T03:05:29.1508003Z   adding: test/allowlist_for_publicAPI.json (deflated 79%)
2022-11-23T03:05:29.1543279Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-11-23T03:05:29.1550989Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2022-11-23T03:05:29.1552129Z   adding: test/.pytorch-slow-tests.json (deflated 73%)
2022-11-23T03:05:29.1563229Z   adding: test/.pytorch-disabled-tests.json (deflated 86%)
2022-11-23T03:05:29.1588319Z ##[group]Run # Remove any previous test reports if they exist
2022-11-23T03:05:29.1588724Z # Remove any previous test reports if they exist
2022-11-23T03:05:29.1589059Z rm -f test-reports-*.zip
2022-11-23T03:05:29.1589426Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2022-11-23T03:05:29.1601432Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:05:29.1601741Z env:
2022-11-23T03:05:29.1601990Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:05:29.1602242Z   GPU_FLAG: --gpus all
2022-11-23T03:05:29.1602597Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:05:29.1603056Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655200056
2022-11-23T03:05:29.1603403Z ##[endgroup]
2022-11-23T03:05:29.1755473Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012138.xml (deflated 41%)
2022-11-23T03:05:29.1756432Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012144.xml (deflated 41%)
2022-11-23T03:05:29.1757323Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012150.xml (deflated 44%)
2022-11-23T03:05:29.1758219Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012152.xml (deflated 44%)
2022-11-23T03:05:29.1759078Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012154.xml (deflated 44%)
2022-11-23T03:05:29.1759971Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012156.xml (deflated 45%)
2022-11-23T03:05:29.1761028Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012201.xml (deflated 41%)
2022-11-23T03:05:29.1761957Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012207.xml (deflated 42%)
2022-11-23T03:05:29.1762821Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012213.xml (deflated 44%)
2022-11-23T03:05:29.1763677Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012215.xml (deflated 44%)
2022-11-23T03:05:29.1764555Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012217.xml (deflated 45%)
2022-11-23T03:05:29.1765426Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20221123012219.xml (deflated 45%)
2022-11-23T03:05:29.1766449Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012233.xml (deflated 41%)
2022-11-23T03:05:29.1767230Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012243.xml (deflated 42%)
2022-11-23T03:05:29.1768043Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012245.xml (deflated 42%)
2022-11-23T03:05:29.1768828Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012252.xml (deflated 42%)
2022-11-23T03:05:29.1769606Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012256.xml (deflated 41%)
2022-11-23T03:05:29.1770373Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012303.xml (deflated 41%)
2022-11-23T03:05:29.1771177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012310.xml (deflated 40%)
2022-11-23T03:05:29.1771955Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012319.xml (deflated 40%)
2022-11-23T03:05:29.1772734Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012327.xml (deflated 41%)
2022-11-23T03:05:29.1773491Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012336.xml (deflated 39%)
2022-11-23T03:05:29.1774271Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012344.xml (deflated 40%)
2022-11-23T03:05:29.1775044Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012353.xml (deflated 40%)
2022-11-23T03:05:29.1775836Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012402.xml (deflated 40%)
2022-11-23T03:05:29.1776630Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012410.xml (deflated 42%)
2022-11-23T03:05:29.1777384Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012415.xml (deflated 41%)
2022-11-23T03:05:29.1778163Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012421.xml (deflated 42%)
2022-11-23T03:05:29.1778938Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012425.xml (deflated 42%)
2022-11-23T03:05:29.1779709Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012434.xml (deflated 41%)
2022-11-23T03:05:29.1780559Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012436.xml (deflated 45%)
2022-11-23T03:05:29.1781355Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012443.xml (deflated 47%)
2022-11-23T03:05:29.1782125Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012450.xml (deflated 48%)
2022-11-23T03:05:29.1782896Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012457.xml (deflated 46%)
2022-11-23T03:05:29.1783655Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012504.xml (deflated 40%)
2022-11-23T03:05:29.1784420Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012510.xml (deflated 41%)
2022-11-23T03:05:29.1785195Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012517.xml (deflated 41%)
2022-11-23T03:05:29.1786058Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012524.xml (deflated 42%)
2022-11-23T03:05:29.1787101Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012530.xml (deflated 41%)
2022-11-23T03:05:29.1787916Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012537.xml (deflated 41%)
2022-11-23T03:05:29.1788690Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012544.xml (deflated 40%)
2022-11-23T03:05:29.1789463Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012551.xml (deflated 41%)
2022-11-23T03:05:29.1790239Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012553.xml (deflated 41%)
2022-11-23T03:05:29.1791034Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012555.xml (deflated 41%)
2022-11-23T03:05:29.1791806Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012602.xml (deflated 42%)
2022-11-23T03:05:29.1792575Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012609.xml (deflated 42%)
2022-11-23T03:05:29.1793346Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012612.xml (deflated 42%)
2022-11-23T03:05:29.1794104Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012614.xml (deflated 42%)
2022-11-23T03:05:29.1794880Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012616.xml (deflated 42%)
2022-11-23T03:05:29.1795662Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012619.xml (deflated 41%)
2022-11-23T03:05:29.1796431Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012626.xml (deflated 40%)
2022-11-23T03:05:29.1797184Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012632.xml (deflated 43%)
2022-11-23T03:05:29.1797958Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012635.xml (deflated 40%)
2022-11-23T03:05:29.1798729Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012642.xml (deflated 40%)
2022-11-23T03:05:29.1799497Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012649.xml (deflated 40%)
2022-11-23T03:05:29.1800363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012655.xml (deflated 40%)
2022-11-23T03:05:29.1801163Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012702.xml (deflated 41%)
2022-11-23T03:05:29.1801932Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012709.xml (deflated 41%)
2022-11-23T03:05:29.1802702Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012716.xml (deflated 42%)
2022-11-23T03:05:29.1803454Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012723.xml (deflated 42%)
2022-11-23T03:05:29.1804227Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012729.xml (deflated 40%)
2022-11-23T03:05:29.1805101Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012736.xml (deflated 40%)
2022-11-23T03:05:29.1805878Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012743.xml (deflated 40%)
2022-11-23T03:05:29.1806631Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012750.xml (deflated 41%)
2022-11-23T03:05:29.1807407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012757.xml (deflated 41%)
2022-11-23T03:05:29.1808175Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012803.xml (deflated 40%)
2022-11-23T03:05:29.1808952Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012810.xml (deflated 41%)
2022-11-23T03:05:29.1809763Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012817.xml (deflated 41%)
2022-11-23T03:05:29.1810588Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012824.xml (deflated 41%)
2022-11-23T03:05:29.1811357Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012831.xml (deflated 40%)
2022-11-23T03:05:29.1812128Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012838.xml (deflated 42%)
2022-11-23T03:05:29.1812901Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012844.xml (deflated 42%)
2022-11-23T03:05:29.1813652Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012851.xml (deflated 42%)
2022-11-23T03:05:29.1814435Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012858.xml (deflated 42%)
2022-11-23T03:05:29.1815212Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012905.xml (deflated 41%)
2022-11-23T03:05:29.1815984Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012912.xml (deflated 41%)
2022-11-23T03:05:29.1816742Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012918.xml (deflated 40%)
2022-11-23T03:05:29.1817510Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012927.xml (deflated 41%)
2022-11-23T03:05:29.1818278Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012936.xml (deflated 40%)
2022-11-23T03:05:29.1819057Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012942.xml (deflated 41%)
2022-11-23T03:05:29.1819890Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012951.xml (deflated 40%)
2022-11-23T03:05:29.1820687Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123012958.xml (deflated 40%)
2022-11-23T03:05:29.1821460Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013004.xml (deflated 40%)
2022-11-23T03:05:29.1822228Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013011.xml (deflated 41%)
2022-11-23T03:05:29.1822983Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013021.xml (deflated 40%)
2022-11-23T03:05:29.1823762Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013031.xml (deflated 41%)
2022-11-23T03:05:29.1824622Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013041.xml (deflated 42%)
2022-11-23T03:05:29.1825393Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013044.xml (deflated 42%)
2022-11-23T03:05:29.1826148Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013046.xml (deflated 42%)
2022-11-23T03:05:29.1827219Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013048.xml (deflated 41%)
2022-11-23T03:05:29.1828015Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013051.xml (deflated 43%)
2022-11-23T03:05:29.1828785Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013053.xml (deflated 42%)
2022-11-23T03:05:29.1829558Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013055.xml (deflated 42%)
2022-11-23T03:05:29.1830341Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013058.xml (deflated 42%)
2022-11-23T03:05:29.1831103Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013100.xml (deflated 43%)
2022-11-23T03:05:29.1831872Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013103.xml (deflated 42%)
2022-11-23T03:05:29.1832629Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013105.xml (deflated 43%)
2022-11-23T03:05:29.1833408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013107.xml (deflated 43%)
2022-11-23T03:05:29.1834183Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013110.xml (deflated 43%)
2022-11-23T03:05:29.1834960Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013112.xml (deflated 42%)
2022-11-23T03:05:29.1835711Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013115.xml (deflated 42%)
2022-11-23T03:05:29.1836483Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013117.xml (deflated 43%)
2022-11-23T03:05:29.1837259Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013119.xml (deflated 43%)
2022-11-23T03:05:29.1838030Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013122.xml (deflated 42%)
2022-11-23T03:05:29.1838887Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013124.xml (deflated 42%)
2022-11-23T03:05:29.1839695Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013127.xml (deflated 42%)
2022-11-23T03:05:29.1840467Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013129.xml (deflated 42%)
2022-11-23T03:05:29.1841243Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013131.xml (deflated 43%)
2022-11-23T03:05:29.1842051Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013134.xml (deflated 42%)
2022-11-23T03:05:29.1842808Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013136.xml (deflated 42%)
2022-11-23T03:05:29.1843582Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013139.xml (deflated 41%)
2022-11-23T03:05:29.1844460Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013148.xml (deflated 42%)
2022-11-23T03:05:29.1845234Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013155.xml (deflated 42%)
2022-11-23T03:05:29.1845991Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013157.xml (deflated 41%)
2022-11-23T03:05:29.1846767Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013205.xml (deflated 41%)
2022-11-23T03:05:29.1847536Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013214.xml (deflated 40%)
2022-11-23T03:05:29.1848304Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013221.xml (deflated 42%)
2022-11-23T03:05:29.1849070Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013228.xml (deflated 42%)
2022-11-23T03:05:29.1849858Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013235.xml (deflated 42%)
2022-11-23T03:05:29.1850627Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013242.xml (deflated 40%)
2022-11-23T03:05:29.1851400Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013249.xml (deflated 42%)
2022-11-23T03:05:29.1852147Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013252.xml (deflated 42%)
2022-11-23T03:05:29.1852921Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013258.xml (deflated 40%)
2022-11-23T03:05:29.1853707Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013305.xml (deflated 41%)
2022-11-23T03:05:29.1854490Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013323.xml (deflated 42%)
2022-11-23T03:05:29.1855434Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013325.xml (deflated 41%)
2022-11-23T03:05:29.1856208Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013327.xml (deflated 41%)
2022-11-23T03:05:29.1856985Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013330.xml (deflated 41%)
2022-11-23T03:05:29.1857749Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013332.xml (deflated 41%)
2022-11-23T03:05:29.1858578Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013335.xml (deflated 41%)
2022-11-23T03:05:29.1859372Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013337.xml (deflated 41%)
2022-11-23T03:05:29.1860146Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013339.xml (deflated 41%)
2022-11-23T03:05:29.1860903Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013342.xml (deflated 41%)
2022-11-23T03:05:29.1861670Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013349.xml (deflated 42%)
2022-11-23T03:05:29.1862419Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013353.xml (deflated 40%)
2022-11-23T03:05:29.1863286Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013400.xml (deflated 42%)
2022-11-23T03:05:29.1864069Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013406.xml (deflated 40%)
2022-11-23T03:05:29.1864840Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013415.xml (deflated 42%)
2022-11-23T03:05:29.1865593Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013419.xml (deflated 42%)
2022-11-23T03:05:29.1866389Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013423.xml (deflated 42%)
2022-11-23T03:05:29.1867388Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013427.xml (deflated 40%)
2022-11-23T03:05:29.1868180Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013436.xml (deflated 40%)
2022-11-23T03:05:29.1868933Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013445.xml (deflated 42%)
2022-11-23T03:05:29.1869710Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013449.xml (deflated 42%)
2022-11-23T03:05:29.1870477Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013453.xml (deflated 40%)
2022-11-23T03:05:29.1871245Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013501.xml (deflated 40%)
2022-11-23T03:05:29.1872000Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013510.xml (deflated 40%)
2022-11-23T03:05:29.1872777Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013518.xml (deflated 40%)
2022-11-23T03:05:29.1873553Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013527.xml (deflated 42%)
2022-11-23T03:05:29.1874331Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013531.xml (deflated 41%)
2022-11-23T03:05:29.1875086Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013538.xml (deflated 42%)
2022-11-23T03:05:29.1875862Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013542.xml (deflated 41%)
2022-11-23T03:05:29.1876637Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013551.xml (deflated 42%)
2022-11-23T03:05:29.1877408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013555.xml (deflated 42%)
2022-11-23T03:05:29.1878260Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013559.xml (deflated 40%)
2022-11-23T03:05:29.1879059Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013609.xml (deflated 40%)
2022-11-23T03:05:29.1879834Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013618.xml (deflated 42%)
2022-11-23T03:05:29.1880603Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013622.xml (deflated 41%)
2022-11-23T03:05:29.1881356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013631.xml (deflated 41%)
2022-11-23T03:05:29.1882135Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013640.xml (deflated 41%)
2022-11-23T03:05:29.1883001Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013649.xml (deflated 41%)
2022-11-23T03:05:29.1883781Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013658.xml (deflated 40%)
2022-11-23T03:05:29.1884535Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013707.xml (deflated 40%)
2022-11-23T03:05:29.1885306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013716.xml (deflated 40%)
2022-11-23T03:05:29.1886079Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013725.xml (deflated 40%)
2022-11-23T03:05:29.1886848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013734.xml (deflated 41%)
2022-11-23T03:05:29.1887614Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013743.xml (deflated 40%)
2022-11-23T03:05:29.1888463Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013752.xml (deflated 41%)
2022-11-23T03:05:29.1889232Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013801.xml (deflated 41%)
2022-11-23T03:05:29.1890003Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013810.xml (deflated 41%)
2022-11-23T03:05:29.1890777Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013819.xml (deflated 42%)
2022-11-23T03:05:29.1891542Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013823.xml (deflated 40%)
2022-11-23T03:05:29.1892313Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013833.xml (deflated 41%)
2022-11-23T03:05:29.1893086Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013841.xml (deflated 41%)
2022-11-23T03:05:29.1893850Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013848.xml (deflated 40%)
2022-11-23T03:05:29.1894621Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013857.xml (deflated 40%)
2022-11-23T03:05:29.1895373Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013905.xml (deflated 40%)
2022-11-23T03:05:29.1896145Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013924.xml (deflated 41%)
2022-11-23T03:05:29.1896993Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013933.xml (deflated 41%)
2022-11-23T03:05:29.1897783Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013942.xml (deflated 41%)
2022-11-23T03:05:29.1898530Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123013951.xml (deflated 40%)
2022-11-23T03:05:29.1899306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014000.xml (deflated 42%)
2022-11-23T03:05:29.1900072Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014004.xml (deflated 43%)
2022-11-23T03:05:29.1900835Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014008.xml (deflated 42%)
2022-11-23T03:05:29.1901608Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014017.xml (deflated 40%)
2022-11-23T03:05:29.1902455Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014026.xml (deflated 42%)
2022-11-23T03:05:29.1903218Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014030.xml (deflated 41%)
2022-11-23T03:05:29.1903984Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014039.xml (deflated 42%)
2022-11-23T03:05:29.1904740Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014043.xml (deflated 41%)
2022-11-23T03:05:29.1905509Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014052.xml (deflated 41%)
2022-11-23T03:05:29.1906277Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014101.xml (deflated 40%)
2022-11-23T03:05:29.1907263Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014109.xml (deflated 42%)
2022-11-23T03:05:29.1908035Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014113.xml (deflated 42%)
2022-11-23T03:05:29.1908806Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014117.xml (deflated 42%)
2022-11-23T03:05:29.1909574Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014121.xml (deflated 40%)
2022-11-23T03:05:29.1910388Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014130.xml (deflated 41%)
2022-11-23T03:05:29.1911140Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014139.xml (deflated 41%)
2022-11-23T03:05:29.1911928Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014146.xml (deflated 41%)
2022-11-23T03:05:29.1912698Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014152.xml (deflated 42%)
2022-11-23T03:05:29.1913466Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014156.xml (deflated 42%)
2022-11-23T03:05:29.1914228Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014201.xml (deflated 40%)
2022-11-23T03:05:29.1914993Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014207.xml (deflated 41%)
2022-11-23T03:05:29.1915763Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014214.xml (deflated 41%)
2022-11-23T03:05:29.1916639Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014221.xml (deflated 42%)
2022-11-23T03:05:29.1917419Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014223.xml (deflated 40%)
2022-11-23T03:05:29.1918188Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014231.xml (deflated 42%)
2022-11-23T03:05:29.1918958Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014237.xml (deflated 40%)
2022-11-23T03:05:29.1919727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014244.xml (deflated 42%)
2022-11-23T03:05:29.1920478Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014248.xml (deflated 41%)
2022-11-23T03:05:29.1921350Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014255.xml (deflated 41%)
2022-11-23T03:05:29.1922118Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014302.xml (deflated 41%)
2022-11-23T03:05:29.1922882Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014309.xml (deflated 40%)
2022-11-23T03:05:29.1923633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014316.xml (deflated 40%)
2022-11-23T03:05:29.1924403Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014323.xml (deflated 40%)
2022-11-23T03:05:29.1925169Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014331.xml (deflated 41%)
2022-11-23T03:05:29.1925953Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014338.xml (deflated 41%)
2022-11-23T03:05:29.1926708Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014345.xml (deflated 40%)
2022-11-23T03:05:29.1927484Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014352.xml (deflated 41%)
2022-11-23T03:05:29.1928257Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014359.xml (deflated 41%)
2022-11-23T03:05:29.1929024Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014424.xml (deflated 41%)
2022-11-23T03:05:29.1929777Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014449.xml (deflated 42%)
2022-11-23T03:05:29.1930558Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014455.xml (deflated 40%)
2022-11-23T03:05:29.1931335Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014504.xml (deflated 40%)
2022-11-23T03:05:29.1932103Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014511.xml (deflated 40%)
2022-11-23T03:05:29.1932871Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014518.xml (deflated 41%)
2022-11-23T03:05:29.1933620Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014525.xml (deflated 42%)
2022-11-23T03:05:29.1934395Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014527.xml (deflated 42%)
2022-11-23T03:05:29.1935155Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014530.xml (deflated 42%)
2022-11-23T03:05:29.1936002Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014532.xml (deflated 41%)
2022-11-23T03:05:29.1936775Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014535.xml (deflated 42%)
2022-11-23T03:05:29.1937537Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014537.xml (deflated 42%)
2022-11-23T03:05:29.1938311Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014539.xml (deflated 42%)
2022-11-23T03:05:29.1939079Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014542.xml (deflated 41%)
2022-11-23T03:05:29.1939841Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014544.xml (deflated 40%)
2022-11-23T03:05:29.1940695Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014551.xml (deflated 40%)
2022-11-23T03:05:29.1941462Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014558.xml (deflated 41%)
2022-11-23T03:05:29.1942240Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014600.xml (deflated 42%)
2022-11-23T03:05:29.1943005Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014603.xml (deflated 42%)
2022-11-23T03:05:29.1943786Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014607.xml (deflated 40%)
2022-11-23T03:05:29.1944567Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014615.xml (deflated 41%)
2022-11-23T03:05:29.1945357Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014625.xml (deflated 40%)
2022-11-23T03:05:29.1946122Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014634.xml (deflated 42%)
2022-11-23T03:05:29.1947095Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014638.xml (deflated 41%)
2022-11-23T03:05:29.1947888Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014643.xml (deflated 40%)
2022-11-23T03:05:29.1948658Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014649.xml (deflated 40%)
2022-11-23T03:05:29.1949419Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014656.xml (deflated 42%)
2022-11-23T03:05:29.1950211Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014700.xml (deflated 40%)
2022-11-23T03:05:29.1950991Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014707.xml (deflated 40%)
2022-11-23T03:05:29.1951761Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014714.xml (deflated 41%)
2022-11-23T03:05:29.1952525Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014721.xml (deflated 40%)
2022-11-23T03:05:29.1953301Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014728.xml (deflated 42%)
2022-11-23T03:05:29.1954296Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014735.xml (deflated 42%)
2022-11-23T03:05:29.1955154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014741.xml (deflated 42%)
2022-11-23T03:05:29.1955933Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014748.xml (deflated 42%)
2022-11-23T03:05:29.1956705Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014755.xml (deflated 40%)
2022-11-23T03:05:29.1957468Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014802.xml (deflated 41%)
2022-11-23T03:05:29.1958316Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014809.xml (deflated 42%)
2022-11-23T03:05:29.1959090Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014811.xml (deflated 41%)
2022-11-23T03:05:29.1959856Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014818.xml (deflated 43%)
2022-11-23T03:05:29.1960723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014820.xml (deflated 43%)
2022-11-23T03:05:29.1961495Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014823.xml (deflated 40%)
2022-11-23T03:05:29.1962267Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014830.xml (deflated 42%)
2022-11-23T03:05:29.1963035Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014832.xml (deflated 42%)
2022-11-23T03:05:29.1963786Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014834.xml (deflated 40%)
2022-11-23T03:05:29.1964557Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014841.xml (deflated 41%)
2022-11-23T03:05:29.1965353Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014848.xml (deflated 41%)
2022-11-23T03:05:29.1966118Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014855.xml (deflated 40%)
2022-11-23T03:05:29.1966885Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014901.xml (deflated 41%)
2022-11-23T03:05:29.1967636Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014904.xml (deflated 41%)
2022-11-23T03:05:29.1968409Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014906.xml (deflated 40%)
2022-11-23T03:05:29.1969172Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014913.xml (deflated 42%)
2022-11-23T03:05:29.1970055Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014920.xml (deflated 40%)
2022-11-23T03:05:29.1970823Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014927.xml (deflated 40%)
2022-11-23T03:05:29.1971589Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014934.xml (deflated 40%)
2022-11-23T03:05:29.1972356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014941.xml (deflated 40%)
2022-11-23T03:05:29.1973120Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014948.xml (deflated 40%)
2022-11-23T03:05:29.1973875Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123014954.xml (deflated 41%)
2022-11-23T03:05:29.1974708Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015001.xml (deflated 41%)
2022-11-23T03:05:29.1975502Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015004.xml (deflated 41%)
2022-11-23T03:05:29.1976268Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015006.xml (deflated 41%)
2022-11-23T03:05:29.1977018Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015008.xml (deflated 41%)
2022-11-23T03:05:29.1977786Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015015.xml (deflated 41%)
2022-11-23T03:05:29.1978550Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015022.xml (deflated 41%)
2022-11-23T03:05:29.1979396Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015029.xml (deflated 40%)
2022-11-23T03:05:29.1980152Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015035.xml (deflated 40%)
2022-11-23T03:05:29.1980924Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015042.xml (deflated 40%)
2022-11-23T03:05:29.1981692Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015051.xml (deflated 41%)
2022-11-23T03:05:29.1982458Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015100.xml (deflated 41%)
2022-11-23T03:05:29.1983211Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015106.xml (deflated 40%)
2022-11-23T03:05:29.1983981Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015115.xml (deflated 41%)
2022-11-23T03:05:29.1984764Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015124.xml (deflated 40%)
2022-11-23T03:05:29.1985543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015137.xml (deflated 40%)
2022-11-23T03:05:29.1986300Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015153.xml (deflated 41%)
2022-11-23T03:05:29.1987262Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015202.xml (deflated 42%)
2022-11-23T03:05:29.1988042Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015205.xml (deflated 42%)
2022-11-23T03:05:29.1988818Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015212.xml (deflated 42%)
2022-11-23T03:05:29.1989579Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015216.xml (deflated 41%)
2022-11-23T03:05:29.1990353Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015223.xml (deflated 41%)
2022-11-23T03:05:29.1991119Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015229.xml (deflated 40%)
2022-11-23T03:05:29.1991888Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015239.xml (deflated 40%)
2022-11-23T03:05:29.1992635Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015247.xml (deflated 41%)
2022-11-23T03:05:29.1993412Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015255.xml (deflated 39%)
2022-11-23T03:05:29.1994272Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015304.xml (deflated 40%)
2022-11-23T03:05:29.1995073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015313.xml (deflated 40%)
2022-11-23T03:05:29.1995829Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015322.xml (deflated 40%)
2022-11-23T03:05:29.1996592Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015330.xml (deflated 42%)
2022-11-23T03:05:29.1997360Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015335.xml (deflated 41%)
2022-11-23T03:05:29.1998125Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015341.xml (deflated 42%)
2022-11-23T03:05:29.1998989Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015346.xml (deflated 42%)
2022-11-23T03:05:29.1999760Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015354.xml (deflated 41%)
2022-11-23T03:05:29.2000535Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015356.xml (deflated 45%)
2022-11-23T03:05:29.2001302Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015403.xml (deflated 47%)
2022-11-23T03:05:29.2002055Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015410.xml (deflated 48%)
2022-11-23T03:05:29.2002831Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015417.xml (deflated 46%)
2022-11-23T03:05:29.2003608Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015424.xml (deflated 40%)
2022-11-23T03:05:29.2004379Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015430.xml (deflated 41%)
2022-11-23T03:05:29.2005129Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015437.xml (deflated 41%)
2022-11-23T03:05:29.2005905Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015444.xml (deflated 41%)
2022-11-23T03:05:29.2006673Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015451.xml (deflated 41%)
2022-11-23T03:05:29.2007437Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015458.xml (deflated 40%)
2022-11-23T03:05:29.2008222Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015505.xml (deflated 41%)
2022-11-23T03:05:29.2008973Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015511.xml (deflated 41%)
2022-11-23T03:05:29.2009787Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015514.xml (deflated 41%)
2022-11-23T03:05:29.2010560Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015516.xml (deflated 41%)
2022-11-23T03:05:29.2011325Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015523.xml (deflated 42%)
2022-11-23T03:05:29.2012082Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015530.xml (deflated 42%)
2022-11-23T03:05:29.2012908Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015532.xml (deflated 42%)
2022-11-23T03:05:29.2013690Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015535.xml (deflated 42%)
2022-11-23T03:05:29.2014461Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015537.xml (deflated 42%)
2022-11-23T03:05:29.2015209Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015539.xml (deflated 40%)
2022-11-23T03:05:29.2015985Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015546.xml (deflated 40%)
2022-11-23T03:05:29.2016746Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015553.xml (deflated 43%)
2022-11-23T03:05:29.2017513Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015555.xml (deflated 40%)
2022-11-23T03:05:29.2018345Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015602.xml (deflated 40%)
2022-11-23T03:05:29.2019113Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015609.xml (deflated 40%)
2022-11-23T03:05:29.2019879Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015616.xml (deflated 40%)
2022-11-23T03:05:29.2020647Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015623.xml (deflated 41%)
2022-11-23T03:05:29.2021401Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015630.xml (deflated 41%)
2022-11-23T03:05:29.2022166Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015636.xml (deflated 42%)
2022-11-23T03:05:29.2022947Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015643.xml (deflated 41%)
2022-11-23T03:05:29.2023722Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015650.xml (deflated 40%)
2022-11-23T03:05:29.2024478Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015657.xml (deflated 41%)
2022-11-23T03:05:29.2025248Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015704.xml (deflated 40%)
2022-11-23T03:05:29.2026012Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015710.xml (deflated 41%)
2022-11-23T03:05:29.2026979Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015717.xml (deflated 41%)
2022-11-23T03:05:29.2027778Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015724.xml (deflated 40%)
2022-11-23T03:05:29.2028553Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015731.xml (deflated 40%)
2022-11-23T03:05:29.2029323Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015738.xml (deflated 40%)
2022-11-23T03:05:29.2030094Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015745.xml (deflated 40%)
2022-11-23T03:05:29.2030851Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015751.xml (deflated 40%)
2022-11-23T03:05:29.2031616Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015758.xml (deflated 42%)
2022-11-23T03:05:29.2032474Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015805.xml (deflated 42%)
2022-11-23T03:05:29.2033265Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015812.xml (deflated 42%)
2022-11-23T03:05:29.2034016Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015819.xml (deflated 42%)
2022-11-23T03:05:29.2034791Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015826.xml (deflated 40%)
2022-11-23T03:05:29.2035559Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015832.xml (deflated 41%)
2022-11-23T03:05:29.2036327Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015839.xml (deflated 40%)
2022-11-23T03:05:29.2037175Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015848.xml (deflated 41%)
2022-11-23T03:05:29.2037951Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015856.xml (deflated 40%)
2022-11-23T03:05:29.2038720Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015903.xml (deflated 41%)
2022-11-23T03:05:29.2039491Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015912.xml (deflated 41%)
2022-11-23T03:05:29.2040244Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015918.xml (deflated 40%)
2022-11-23T03:05:29.2041011Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015925.xml (deflated 41%)
2022-11-23T03:05:29.2041789Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015932.xml (deflated 40%)
2022-11-23T03:05:29.2042565Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015942.xml (deflated 40%)
2022-11-23T03:05:29.2043314Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123015952.xml (deflated 41%)
2022-11-23T03:05:29.2044086Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020002.xml (deflated 43%)
2022-11-23T03:05:29.2044855Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020004.xml (deflated 42%)
2022-11-23T03:05:29.2045621Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020007.xml (deflated 42%)
2022-11-23T03:05:29.2046377Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020009.xml (deflated 42%)
2022-11-23T03:05:29.2047153Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020012.xml (deflated 43%)
2022-11-23T03:05:29.2047921Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020014.xml (deflated 42%)
2022-11-23T03:05:29.2048690Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020016.xml (deflated 43%)
2022-11-23T03:05:29.2049455Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020019.xml (deflated 42%)
2022-11-23T03:05:29.2050203Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020021.xml (deflated 43%)
2022-11-23T03:05:29.2050977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020024.xml (deflated 42%)
2022-11-23T03:05:29.2051810Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020026.xml (deflated 43%)
2022-11-23T03:05:29.2052590Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020028.xml (deflated 42%)
2022-11-23T03:05:29.2053344Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020031.xml (deflated 43%)
2022-11-23T03:05:29.2054119Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020033.xml (deflated 42%)
2022-11-23T03:05:29.2054939Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020036.xml (deflated 42%)
2022-11-23T03:05:29.2055898Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020038.xml (deflated 43%)
2022-11-23T03:05:29.2056732Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020040.xml (deflated 43%)
2022-11-23T03:05:29.2057502Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020043.xml (deflated 42%)
2022-11-23T03:05:29.2058267Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020045.xml (deflated 42%)
2022-11-23T03:05:29.2059035Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020047.xml (deflated 42%)
2022-11-23T03:05:29.2059793Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020050.xml (deflated 42%)
2022-11-23T03:05:29.2060563Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020052.xml (deflated 42%)
2022-11-23T03:05:29.2061337Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020055.xml (deflated 42%)
2022-11-23T03:05:29.2062107Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020057.xml (deflated 43%)
2022-11-23T03:05:29.2062854Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020059.xml (deflated 41%)
2022-11-23T03:05:29.2063620Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020109.xml (deflated 42%)
2022-11-23T03:05:29.2064385Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020116.xml (deflated 42%)
2022-11-23T03:05:29.2065149Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020118.xml (deflated 41%)
2022-11-23T03:05:29.2066018Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020125.xml (deflated 41%)
2022-11-23T03:05:29.2066955Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020135.xml (deflated 40%)
2022-11-23T03:05:29.2067727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020142.xml (deflated 42%)
2022-11-23T03:05:29.2068499Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020149.xml (deflated 42%)
2022-11-23T03:05:29.2069261Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020156.xml (deflated 42%)
2022-11-23T03:05:29.2070023Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020203.xml (deflated 41%)
2022-11-23T03:05:29.2070879Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020210.xml (deflated 41%)
2022-11-23T03:05:29.2071682Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020218.xml (deflated 42%)
2022-11-23T03:05:29.2072447Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020225.xml (deflated 41%)
2022-11-23T03:05:29.2073208Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020232.xml (deflated 41%)
2022-11-23T03:05:29.2073962Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020238.xml (deflated 41%)
2022-11-23T03:05:29.2074727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020241.xml (deflated 41%)
2022-11-23T03:05:29.2075492Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020243.xml (deflated 41%)
2022-11-23T03:05:29.2076354Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020246.xml (deflated 41%)
2022-11-23T03:05:29.2077112Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020248.xml (deflated 41%)
2022-11-23T03:05:29.2077890Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020250.xml (deflated 41%)
2022-11-23T03:05:29.2078659Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020253.xml (deflated 41%)
2022-11-23T03:05:29.2079425Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020255.xml (deflated 42%)
2022-11-23T03:05:29.2080172Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020258.xml (deflated 41%)
2022-11-23T03:05:29.2080956Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020304.xml (deflated 42%)
2022-11-23T03:05:29.2081728Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020309.xml (deflated 40%)
2022-11-23T03:05:29.2082492Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020315.xml (deflated 42%)
2022-11-23T03:05:29.2083241Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020322.xml (deflated 40%)
2022-11-23T03:05:29.2084009Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020330.xml (deflated 42%)
2022-11-23T03:05:29.2084779Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020335.xml (deflated 42%)
2022-11-23T03:05:29.2085556Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020339.xml (deflated 42%)
2022-11-23T03:05:29.2086325Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020343.xml (deflated 40%)
2022-11-23T03:05:29.2087079Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020352.xml (deflated 40%)
2022-11-23T03:05:29.2087850Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020401.xml (deflated 42%)
2022-11-23T03:05:29.2088619Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020405.xml (deflated 42%)
2022-11-23T03:05:29.2089389Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020409.xml (deflated 40%)
2022-11-23T03:05:29.2090202Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020417.xml (deflated 40%)
2022-11-23T03:05:29.2090984Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020426.xml (deflated 40%)
2022-11-23T03:05:29.2091750Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020434.xml (deflated 41%)
2022-11-23T03:05:29.2092514Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020443.xml (deflated 42%)
2022-11-23T03:05:29.2093266Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020447.xml (deflated 41%)
2022-11-23T03:05:29.2094034Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020454.xml (deflated 42%)
2022-11-23T03:05:29.2094871Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020458.xml (deflated 41%)
2022-11-23T03:05:29.2095645Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020507.xml (deflated 42%)
2022-11-23T03:05:29.2096394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020511.xml (deflated 42%)
2022-11-23T03:05:29.2097156Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020516.xml (deflated 40%)
2022-11-23T03:05:29.2097918Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020525.xml (deflated 40%)
2022-11-23T03:05:29.2098766Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020534.xml (deflated 42%)
2022-11-23T03:05:29.2128927Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020538.xml (deflated 41%)
2022-11-23T03:05:29.2129828Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020547.xml (deflated 41%)
2022-11-23T03:05:29.2130630Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020556.xml (deflated 41%)
2022-11-23T03:05:29.2131409Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020605.xml (deflated 41%)
2022-11-23T03:05:29.2132192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020614.xml (deflated 41%)
2022-11-23T03:05:29.2132985Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020623.xml (deflated 40%)
2022-11-23T03:05:29.2133775Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020632.xml (deflated 41%)
2022-11-23T03:05:29.2134546Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020641.xml (deflated 41%)
2022-11-23T03:05:29.2135336Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020650.xml (deflated 41%)
2022-11-23T03:05:29.2136125Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020659.xml (deflated 40%)
2022-11-23T03:05:29.2136906Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020708.xml (deflated 41%)
2022-11-23T03:05:29.2137669Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020717.xml (deflated 41%)
2022-11-23T03:05:29.2138452Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020726.xml (deflated 41%)
2022-11-23T03:05:29.2139380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020735.xml (deflated 42%)
2022-11-23T03:05:29.2140196Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020739.xml (deflated 40%)
2022-11-23T03:05:29.2140955Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020748.xml (deflated 40%)
2022-11-23T03:05:29.2141736Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020757.xml (deflated 40%)
2022-11-23T03:05:29.2142512Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020804.xml (deflated 40%)
2022-11-23T03:05:29.2143283Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020813.xml (deflated 40%)
2022-11-23T03:05:29.2144178Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020821.xml (deflated 40%)
2022-11-23T03:05:29.2144946Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020839.xml (deflated 41%)
2022-11-23T03:05:29.2145729Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020848.xml (deflated 41%)
2022-11-23T03:05:29.2146508Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020857.xml (deflated 41%)
2022-11-23T03:05:29.2147689Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020906.xml (deflated 40%)
2022-11-23T03:05:29.2148456Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020915.xml (deflated 42%)
2022-11-23T03:05:29.2149245Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020919.xml (deflated 43%)
2022-11-23T03:05:29.2150023Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020923.xml (deflated 42%)
2022-11-23T03:05:29.2150795Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020933.xml (deflated 41%)
2022-11-23T03:05:29.2151552Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020941.xml (deflated 42%)
2022-11-23T03:05:29.2152331Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020945.xml (deflated 41%)
2022-11-23T03:05:29.2153295Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020954.xml (deflated 42%)
2022-11-23T03:05:29.2154068Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123020958.xml (deflated 41%)
2022-11-23T03:05:29.2154831Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021007.xml (deflated 41%)
2022-11-23T03:05:29.2155609Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021016.xml (deflated 40%)
2022-11-23T03:05:29.2156386Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021024.xml (deflated 42%)
2022-11-23T03:05:29.2157153Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021028.xml (deflated 42%)
2022-11-23T03:05:29.2157909Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021032.xml (deflated 42%)
2022-11-23T03:05:29.2158774Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021036.xml (deflated 41%)
2022-11-23T03:05:29.2159570Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021045.xml (deflated 42%)
2022-11-23T03:05:29.2160347Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021053.xml (deflated 40%)
2022-11-23T03:05:29.2161102Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021100.xml (deflated 42%)
2022-11-23T03:05:29.2161869Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021107.xml (deflated 42%)
2022-11-23T03:05:29.2162651Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021111.xml (deflated 42%)
2022-11-23T03:05:29.2163426Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021115.xml (deflated 40%)
2022-11-23T03:05:29.2164281Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021122.xml (deflated 41%)
2022-11-23T03:05:29.2165061Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021129.xml (deflated 41%)
2022-11-23T03:05:29.2165834Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021136.xml (deflated 42%)
2022-11-23T03:05:29.2166608Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021138.xml (deflated 40%)
2022-11-23T03:05:29.2167368Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021145.xml (deflated 42%)
2022-11-23T03:05:29.2168150Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021152.xml (deflated 40%)
2022-11-23T03:05:29.2168942Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021159.xml (deflated 42%)
2022-11-23T03:05:29.2169713Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021203.xml (deflated 41%)
2022-11-23T03:05:29.2170471Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021210.xml (deflated 41%)
2022-11-23T03:05:29.2171250Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021217.xml (deflated 41%)
2022-11-23T03:05:29.2172021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021224.xml (deflated 40%)
2022-11-23T03:05:29.2172792Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021231.xml (deflated 40%)
2022-11-23T03:05:29.2173561Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021237.xml (deflated 40%)
2022-11-23T03:05:29.2174351Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021246.xml (deflated 41%)
2022-11-23T03:05:29.2175111Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021253.xml (deflated 41%)
2022-11-23T03:05:29.2175882Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021300.xml (deflated 40%)
2022-11-23T03:05:29.2176639Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021306.xml (deflated 41%)
2022-11-23T03:05:29.2177416Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021313.xml (deflated 41%)
2022-11-23T03:05:29.2178272Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021338.xml (deflated 41%)
2022-11-23T03:05:29.2179063Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021403.xml (deflated 42%)
2022-11-23T03:05:29.2179818Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021410.xml (deflated 40%)
2022-11-23T03:05:29.2180596Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021419.xml (deflated 40%)
2022-11-23T03:05:29.2181652Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021426.xml (deflated 40%)
2022-11-23T03:05:29.2182440Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021433.xml (deflated 41%)
2022-11-23T03:05:29.2183287Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021439.xml (deflated 42%)
2022-11-23T03:05:29.2184067Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021442.xml (deflated 42%)
2022-11-23T03:05:29.2184842Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021444.xml (deflated 42%)
2022-11-23T03:05:29.2185614Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021447.xml (deflated 41%)
2022-11-23T03:05:29.2186388Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021449.xml (deflated 42%)
2022-11-23T03:05:29.2187344Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021451.xml (deflated 42%)
2022-11-23T03:05:29.2188130Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021454.xml (deflated 42%)
2022-11-23T03:05:29.2188908Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021456.xml (deflated 41%)
2022-11-23T03:05:29.2189684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021458.xml (deflated 40%)
2022-11-23T03:05:29.2190445Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021505.xml (deflated 40%)
2022-11-23T03:05:29.2191216Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021512.xml (deflated 42%)
2022-11-23T03:05:29.2191988Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021515.xml (deflated 42%)
2022-11-23T03:05:29.2192763Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021517.xml (deflated 42%)
2022-11-23T03:05:29.2193532Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021521.xml (deflated 40%)
2022-11-23T03:05:29.2194305Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021530.xml (deflated 41%)
2022-11-23T03:05:29.2195077Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021539.xml (deflated 40%)
2022-11-23T03:05:29.2195849Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021549.xml (deflated 42%)
2022-11-23T03:05:29.2196612Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021553.xml (deflated 41%)
2022-11-23T03:05:29.2197385Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021557.xml (deflated 40%)
2022-11-23T03:05:29.2198225Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021604.xml (deflated 40%)
2022-11-23T03:05:29.2199007Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021610.xml (deflated 42%)
2022-11-23T03:05:29.2199753Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021615.xml (deflated 40%)
2022-11-23T03:05:29.2200522Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021621.xml (deflated 40%)
2022-11-23T03:05:29.2201269Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021628.xml (deflated 40%)
2022-11-23T03:05:29.2202016Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021635.xml (deflated 40%)
2022-11-23T03:05:29.2202856Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021642.xml (deflated 42%)
2022-11-23T03:05:29.2203608Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021649.xml (deflated 42%)
2022-11-23T03:05:29.2204374Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021656.xml (deflated 42%)
2022-11-23T03:05:29.2205139Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021702.xml (deflated 42%)
2022-11-23T03:05:29.2205894Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021709.xml (deflated 41%)
2022-11-23T03:05:29.2206665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021716.xml (deflated 41%)
2022-11-23T03:05:29.2207446Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021723.xml (deflated 42%)
2022-11-23T03:05:29.2208217Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021725.xml (deflated 41%)
2022-11-23T03:05:29.2208972Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021732.xml (deflated 43%)
2022-11-23T03:05:29.2209744Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021734.xml (deflated 43%)
2022-11-23T03:05:29.2210554Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021737.xml (deflated 41%)
2022-11-23T03:05:29.2211323Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021743.xml (deflated 42%)
2022-11-23T03:05:29.2212087Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021746.xml (deflated 42%)
2022-11-23T03:05:29.2212858Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021748.xml (deflated 41%)
2022-11-23T03:05:29.2213632Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021755.xml (deflated 41%)
2022-11-23T03:05:29.2214405Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021802.xml (deflated 41%)
2022-11-23T03:05:29.2215162Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021809.xml (deflated 40%)
2022-11-23T03:05:29.2215932Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021816.xml (deflated 42%)
2022-11-23T03:05:29.2216753Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021818.xml (deflated 41%)
2022-11-23T03:05:29.2217544Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021820.xml (deflated 40%)
2022-11-23T03:05:29.2218293Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021827.xml (deflated 42%)
2022-11-23T03:05:29.2219064Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021834.xml (deflated 41%)
2022-11-23T03:05:29.2219832Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021841.xml (deflated 41%)
2022-11-23T03:05:29.2220594Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021848.xml (deflated 40%)
2022-11-23T03:05:29.2221345Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021854.xml (deflated 40%)
2022-11-23T03:05:29.2222191Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021901.xml (deflated 40%)
2022-11-23T03:05:29.2222945Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021908.xml (deflated 41%)
2022-11-23T03:05:29.2223695Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021915.xml (deflated 41%)
2022-11-23T03:05:29.2224436Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021917.xml (deflated 41%)
2022-11-23T03:05:29.2225211Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021919.xml (deflated 41%)
2022-11-23T03:05:29.2225981Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021922.xml (deflated 40%)
2022-11-23T03:05:29.2226913Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021929.xml (deflated 41%)
2022-11-23T03:05:29.2227707Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021935.xml (deflated 40%)
2022-11-23T03:05:29.2228481Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021942.xml (deflated 40%)
2022-11-23T03:05:29.2229254Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021949.xml (deflated 40%)
2022-11-23T03:05:29.2230021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123021956.xml (deflated 41%)
2022-11-23T03:05:29.2230771Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022004.xml (deflated 41%)
2022-11-23T03:05:29.2231554Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022013.xml (deflated 41%)
2022-11-23T03:05:29.2232332Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022020.xml (deflated 40%)
2022-11-23T03:05:29.2233098Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022028.xml (deflated 41%)
2022-11-23T03:05:29.2233852Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022037.xml (deflated 40%)
2022-11-23T03:05:29.2234604Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221123022050.xml (deflated 41%)
2022-11-23T03:05:29.2235352Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_traverse/TEST-TestTraverse-20221123022138.xml (deflated 77%)
2022-11-23T03:05:29.2236153Z   adding: test/test-reports/python-unittest/distributed.test_nccl/TEST-TestNCCLCUDA-20221123022146.xml (deflated 83%)
2022-11-23T03:05:29.2236909Z   adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20221123022149.xml (deflated 63%)
2022-11-23T03:05:29.2237664Z   adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20221123022153.xml (deflated 64%)
2022-11-23T03:05:29.2238424Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20221123022157.xml (deflated 54%)
2022-11-23T03:05:29.2239212Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20221123022201.xml (deflated 68%)
2022-11-23T03:05:29.2240091Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20221123022206.xml (deflated 44%)
2022-11-23T03:05:29.2241098Z   adding: test/test-reports/python-unittest/distributed._tensor.parallel.test_view_sharding_dim_change/TEST-TPViewShardingDimChangeTest-20221123022212.xml (deflated 43%)
2022-11-23T03:05:29.2241968Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20221123022220.xml (deflated 66%)
2022-11-23T03:05:29.2242773Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20221123022220.xml (deflated 69%)
2022-11-23T03:05:29.2243633Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20221123022220.xml (deflated 66%)
2022-11-23T03:05:29.2244483Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20221123022229.xml (deflated 47%)
2022-11-23T03:05:29.2245293Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20221123022237.xml (deflated 61%)
2022-11-23T03:05:29.2246121Z   adding: test/test-reports/python-unittest/distributed._tensor.parallel.test_2d_parallel/TEST-Test2dParallelIntegration-20221123022246.xml (deflated 59%)
2022-11-23T03:05:29.2246887Z   adding: test/test-reports/python-unittest/distributed._tensor.test_view_ops/TEST-TestViewOps-20221123022255.xml (deflated 51%)
2022-11-23T03:05:29.2247667Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20221123022303.xml (deflated 68%)
2022-11-23T03:05:29.2248507Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20221123022315.xml (deflated 54%)
2022-11-23T03:05:29.2249350Z   adding: test/test-reports/python-unittest/distributed._tensor.parallel.test_parallelize_api/TEST-TensorParallelAPITests-20221123022328.xml (deflated 73%)
2022-11-23T03:05:29.2250180Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20221123022341.xml (deflated 76%)
2022-11-23T03:05:29.2250942Z   adding: test/test-reports/python-unittest/distributed._tensor.test_api/TEST-DTensorAPITest-20221123022358.xml (deflated 75%)
2022-11-23T03:05:29.2251730Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20221123022414.xml (deflated 43%)
2022-11-23T03:05:29.2252597Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20221123022414.xml (deflated 43%)
2022-11-23T03:05:29.2253434Z   adding: test/test-reports/python-unittest/distributed.test_c10d_object_collectives/TEST-TestObjectCollectives-20221123022430.xml (deflated 68%)
2022-11-23T03:05:29.2254274Z   adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributed-20221123022450.xml (deflated 87%)
2022-11-23T03:05:29.2255117Z   adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributedMultiProc-20221123022450.xml (deflated 73%)
2022-11-23T03:05:29.2255941Z   adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPInitialization-20221123022509.xml (deflated 68%)
2022-11-23T03:05:29.2256704Z   adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPRuntime-20221123022509.xml (deflated 39%)
2022-11-23T03:05:29.2257501Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20221123022533.xml (deflated 75%)
2022-11-23T03:05:29.2258334Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20221123022559.xml (deflated 55%)
2022-11-23T03:05:29.2259198Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20221123022559.xml (deflated 81%)
2022-11-23T03:05:29.2260120Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20221123022559.xml (deflated 80%)
2022-11-23T03:05:29.2260931Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20221123022559.xml (deflated 64%)
2022-11-23T03:05:29.2261714Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20221123022628.xml (deflated 86%)
2022-11-23T03:05:29.2262606Z   adding: test/test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20221123022703.xml (deflated 79%)
2022-11-23T03:05:29.2263453Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_exec_order/TEST-TestFSDPExecOrder-20221123022739.xml (deflated 83%)
2022-11-23T03:05:29.2264182Z   adding: test/test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20221123022819.xml (deflated 75%)
2022-11-23T03:05:29.2264905Z   adding: test/test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20221123022819.xml (deflated 75%)
2022-11-23T03:05:29.2265616Z   adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20221123022903.xml (deflated 80%)
2022-11-23T03:05:29.2266324Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20221123022952.xml (deflated 93%)
2022-11-23T03:05:29.2267240Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_misc/TEST-TestFSDPMisc-20221123023050.xml (deflated 77%)
2022-11-23T03:05:29.2268010Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20221123023200.xml (deflated 90%)
2022-11-23T03:05:29.2268787Z   adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20221123023316.xml (deflated 84%)
2022-11-23T03:05:29.2269540Z   adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20221123023316.xml (deflated 84%)
2022-11-23T03:05:29.2270353Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20221123023541.xml (deflated 54%)
2022-11-23T03:05:29.2271275Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20221123023541.xml (deflated 83%)
2022-11-23T03:05:29.2272211Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20221123023541.xml (deflated 45%)
2022-11-23T03:05:29.2273127Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20221123023541.xml (deflated 76%)
2022-11-23T03:05:29.2274022Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20221123023541.xml (deflated 64%)
2022-11-23T03:05:29.2274982Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20221123023846.xml (deflated 92%)
2022-11-23T03:05:29.2275877Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20221123023846.xml (deflated 64%)
2022-11-23T03:05:29.2276773Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20221123024405.xml (deflated 41%)
2022-11-23T03:05:29.2277652Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123024412.xml (deflated 41%)
2022-11-23T03:05:29.2278540Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123024422.xml (deflated 41%)
2022-11-23T03:05:29.2279427Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20221123024431.xml (deflated 41%)
2022-11-23T03:05:29.2280422Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024441.xml (deflated 41%)
2022-11-23T03:05:29.2281289Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024449.xml (deflated 41%)
2022-11-23T03:05:29.2282170Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024458.xml (deflated 41%)
2022-11-23T03:05:29.2283065Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20221123024506.xml (deflated 41%)
2022-11-23T03:05:29.2283913Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20221123024514.xml (deflated 40%)
2022-11-23T03:05:29.2284765Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024526.xml (deflated 40%)
2022-11-23T03:05:29.2285611Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024533.xml (deflated 40%)
2022-11-23T03:05:29.2286471Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024540.xml (deflated 40%)
2022-11-23T03:05:29.2287329Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024547.xml (deflated 40%)
2022-11-23T03:05:29.2288181Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024554.xml (deflated 40%)
2022-11-23T03:05:29.2289027Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024601.xml (deflated 40%)
2022-11-23T03:05:29.2289897Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024608.xml (deflated 40%)
2022-11-23T03:05:29.2290742Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20221123024615.xml (deflated 40%)
2022-11-23T03:05:29.2291661Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024622.xml (deflated 42%)
2022-11-23T03:05:29.2292601Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024636.xml (deflated 42%)
2022-11-23T03:05:29.2293559Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024651.xml (deflated 42%)
2022-11-23T03:05:29.2294582Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024705.xml (deflated 43%)
2022-11-23T03:05:29.2295551Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024718.xml (deflated 43%)
2022-11-23T03:05:29.2296504Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024730.xml (deflated 43%)
2022-11-23T03:05:29.2297440Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024743.xml (deflated 43%)
2022-11-23T03:05:29.2298394Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024755.xml (deflated 43%)
2022-11-23T03:05:29.2299346Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024808.xml (deflated 43%)
2022-11-23T03:05:29.2300365Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024820.xml (deflated 43%)
2022-11-23T03:05:29.2301300Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024827.xml (deflated 43%)
2022-11-23T03:05:29.2302245Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024835.xml (deflated 43%)
2022-11-23T03:05:29.2303197Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024842.xml (deflated 43%)
2022-11-23T03:05:29.2304146Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024849.xml (deflated 43%)
2022-11-23T03:05:29.2305104Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024858.xml (deflated 43%)
2022-11-23T03:05:29.2306036Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024906.xml (deflated 42%)
2022-11-23T03:05:29.2307174Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024920.xml (deflated 43%)
2022-11-23T03:05:29.2308145Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024925.xml (deflated 43%)
2022-11-23T03:05:29.2309087Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123024946.xml (deflated 43%)
2022-11-23T03:05:29.2310063Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025002.xml (deflated 43%)
2022-11-23T03:05:29.2311015Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025017.xml (deflated 43%)
2022-11-23T03:05:29.2311957Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025024.xml (deflated 42%)
2022-11-23T03:05:29.2312910Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025035.xml (deflated 42%)
2022-11-23T03:05:29.2313850Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025045.xml (deflated 42%)
2022-11-23T03:05:29.2314779Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025056.xml (deflated 43%)
2022-11-23T03:05:29.2315812Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025100.xml (deflated 43%)
2022-11-23T03:05:29.2316780Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025113.xml (deflated 43%)
2022-11-23T03:05:29.2317725Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025126.xml (deflated 42%)
2022-11-23T03:05:29.2318655Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025139.xml (deflated 43%)
2022-11-23T03:05:29.2319603Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025152.xml (deflated 43%)
2022-11-23T03:05:29.2320643Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025205.xml (deflated 42%)
2022-11-23T03:05:29.2321590Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025218.xml (deflated 42%)
2022-11-23T03:05:29.2322513Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025231.xml (deflated 43%)
2022-11-23T03:05:29.2323454Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025244.xml (deflated 43%)
2022-11-23T03:05:29.2324401Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025257.xml (deflated 43%)
2022-11-23T03:05:29.2325362Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025310.xml (deflated 43%)
2022-11-23T03:05:29.2326309Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025323.xml (deflated 42%)
2022-11-23T03:05:29.2327234Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025336.xml (deflated 42%)
2022-11-23T03:05:29.2328183Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025349.xml (deflated 42%)
2022-11-23T03:05:29.2329130Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025402.xml (deflated 42%)
2022-11-23T03:05:29.2330073Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025416.xml (deflated 42%)
2022-11-23T03:05:29.2331009Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025428.xml (deflated 42%)
2022-11-23T03:05:29.2331955Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025439.xml (deflated 43%)
2022-11-23T03:05:29.2332897Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025452.xml (deflated 42%)
2022-11-23T03:05:29.2333845Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025503.xml (deflated 42%)
2022-11-23T03:05:29.2334785Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025513.xml (deflated 43%)
2022-11-23T03:05:29.2335774Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025526.xml (deflated 43%)
2022-11-23T03:05:29.2336734Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025539.xml (deflated 43%)
2022-11-23T03:05:29.2337686Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025547.xml (deflated 43%)
2022-11-23T03:05:29.2338620Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025554.xml (deflated 43%)
2022-11-23T03:05:29.2339542Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025601.xml (deflated 42%)
2022-11-23T03:05:29.2340557Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025608.xml (deflated 42%)
2022-11-23T03:05:29.2341502Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025618.xml (deflated 42%)
2022-11-23T03:05:29.2342448Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025627.xml (deflated 42%)
2022-11-23T03:05:29.2343391Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025636.xml (deflated 42%)
2022-11-23T03:05:29.2344317Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025646.xml (deflated 42%)
2022-11-23T03:05:29.2345258Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025655.xml (deflated 42%)
2022-11-23T03:05:29.2346213Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025705.xml (deflated 42%)
2022-11-23T03:05:29.2347346Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025714.xml (deflated 42%)
2022-11-23T03:05:29.2348284Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025727.xml (deflated 42%)
2022-11-23T03:05:29.2349220Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025740.xml (deflated 43%)
2022-11-23T03:05:29.2350168Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025747.xml (deflated 43%)
2022-11-23T03:05:29.2351128Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025800.xml (deflated 43%)
2022-11-23T03:05:29.2352057Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025808.xml (deflated 43%)
2022-11-23T03:05:29.2353001Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025815.xml (deflated 42%)
2022-11-23T03:05:29.2353947Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025822.xml (deflated 43%)
2022-11-23T03:05:29.2354948Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025833.xml (deflated 42%)
2022-11-23T03:05:29.2355982Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025840.xml (deflated 42%)
2022-11-23T03:05:29.2356929Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025847.xml (deflated 42%)
2022-11-23T03:05:29.2357882Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025900.xml (deflated 42%)
2022-11-23T03:05:29.2358828Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025915.xml (deflated 42%)
2022-11-23T03:05:29.2359778Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025931.xml (deflated 42%)
2022-11-23T03:05:29.2360734Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123025944.xml (deflated 42%)
2022-11-23T03:05:29.2361800Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030004.xml (deflated 42%)
2022-11-23T03:05:29.2362788Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030027.xml (deflated 43%)
2022-11-23T03:05:29.2363724Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030032.xml (deflated 42%)
2022-11-23T03:05:29.2364671Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030055.xml (deflated 42%)
2022-11-23T03:05:29.2365606Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030115.xml (deflated 42%)
2022-11-23T03:05:29.2366544Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030134.xml (deflated 42%)
2022-11-23T03:05:29.2367485Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030154.xml (deflated 42%)
2022-11-23T03:05:29.2368435Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030213.xml (deflated 42%)
2022-11-23T03:05:29.2369381Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030232.xml (deflated 42%)
2022-11-23T03:05:29.2370328Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030253.xml (deflated 42%)
2022-11-23T03:05:29.2371269Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030316.xml (deflated 42%)
2022-11-23T03:05:29.2372216Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030336.xml (deflated 42%)
2022-11-23T03:05:29.2373164Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030359.xml (deflated 42%)
2022-11-23T03:05:29.2374107Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20221123030410.xml (deflated 43%)
2022-11-23T03:05:29.2375057Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123030422.xml (deflated 44%)
2022-11-23T03:05:29.2376098Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123030429.xml (deflated 44%)
2022-11-23T03:05:29.2377112Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20221123030436.xml (deflated 44%)
2022-11-23T03:05:29.2377874Z   adding: test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml (deflated 72%)
2022-11-23T03:05:29.2378439Z   adding: test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml (deflated 72%)
2022-11-23T03:05:29.2379017Z   adding: test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml (deflated 81%)
2022-11-23T03:05:29.2379631Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml (deflated 81%)
2022-11-23T03:05:29.2380252Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml (deflated 79%)
2022-11-23T03:05:29.2380890Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml (deflated 67%)
2022-11-23T03:05:29.2381541Z   adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%)
2022-11-23T03:05:29.2402194Z ##[group]Run # Remove any previous test reports if they exist
2022-11-23T03:05:29.2402558Z # Remove any previous test reports if they exist
2022-11-23T03:05:29.2402878Z rm -f usage-log-*.zip
2022-11-23T03:05:29.2403257Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-11-23T03:05:29.2403643Z # so check to see if the file exists first
2022-11-23T03:05:29.2403936Z if [ -f 'usage_log.txt' ]; then
2022-11-23T03:05:29.2404273Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-11-23T03:05:29.2404569Z fi
2022-11-23T03:05:29.2416258Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:05:29.2416562Z env:
2022-11-23T03:05:29.2416804Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:05:29.2417060Z   GPU_FLAG: --gpus all
2022-11-23T03:05:29.2417412Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:05:29.2417874Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655200056
2022-11-23T03:05:29.2418227Z ##[endgroup]
2022-11-23T03:05:29.3357410Z   adding: usage_log.txt
2022-11-23T03:05:29.3357773Z 	zip warning:  file size changed while zipping usage_log.txt
2022-11-23T03:05:29.3358060Z  (deflated 95%)
2022-11-23T03:05:29.3403142Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T03:05:29.3403434Z with:
2022-11-23T03:05:29.3403715Z   s3-prefix: pytorch/pytorch/3528293562/1/artifact

2022-11-23T03:05:29.3404013Z   retention-days: 14
2022-11-23T03:05:29.3404261Z   if-no-files-found: warn
2022-11-23T03:05:29.3404533Z   path: test-jsons-*.zip
2022-11-23T03:05:29.3404787Z   name: artifact
2022-11-23T03:05:29.3405019Z   s3-bucket: gha-artifacts
2022-11-23T03:05:29.3405280Z   region: us-east-1
2022-11-23T03:05:29.3405512Z env:
2022-11-23T03:05:29.3405735Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:05:29.3405996Z   GPU_FLAG: --gpus all
2022-11-23T03:05:29.3406346Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:05:29.3406660Z ##[endgroup]
2022-11-23T03:05:29.7817978Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T03:05:29.7818765Z With the provided path, there will be 1 file uploaded
2022-11-23T03:05:29.7819607Z Uploading to s3 prefix: pytorch/pytorch/3528293562/1/artifact
2022-11-23T03:05:29.7830938Z Starting upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655200056.zip
2022-11-23T03:05:29.9097233Z Finished upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655200056.zip
2022-11-23T03:05:29.9268714Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T03:05:29.9269007Z with:
2022-11-23T03:05:29.9269289Z   s3-prefix: pytorch/pytorch/3528293562/1/artifact

2022-11-23T03:05:29.9269589Z   retention-days: 14
2022-11-23T03:05:29.9269846Z   if-no-files-found: error
2022-11-23T03:05:29.9270136Z   path: test-reports-*.zip
2022-11-23T03:05:29.9270501Z   name: artifact
2022-11-23T03:05:29.9270747Z   s3-bucket: gha-artifacts
2022-11-23T03:05:29.9271007Z   region: us-east-1
2022-11-23T03:05:29.9271239Z env:
2022-11-23T03:05:29.9271457Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:05:29.9271721Z   GPU_FLAG: --gpus all
2022-11-23T03:05:29.9272064Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:05:29.9272374Z ##[endgroup]
2022-11-23T03:05:30.3670069Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T03:05:30.3670872Z With the provided path, there will be 1 file uploaded
2022-11-23T03:05:30.3671253Z Uploading to s3 prefix: pytorch/pytorch/3528293562/1/artifact
2022-11-23T03:05:30.3682277Z Starting upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655200056.zip
2022-11-23T03:05:30.7430703Z Finished upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655200056.zip
2022-11-23T03:05:30.7603211Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-11-23T03:05:30.7603497Z with:
2022-11-23T03:05:30.7603788Z   s3-prefix: pytorch/pytorch/3528293562/1/artifact

2022-11-23T03:05:30.7604092Z   retention-days: 14
2022-11-23T03:05:30.7604345Z   if-no-files-found: ignore
2022-11-23T03:05:30.7604620Z   path: usage-log-*.zip
2022-11-23T03:05:30.7604872Z   name: artifact
2022-11-23T03:05:30.7605105Z   s3-bucket: gha-artifacts
2022-11-23T03:05:30.7605366Z   region: us-east-1
2022-11-23T03:05:30.7605593Z env:
2022-11-23T03:05:30.7605811Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:05:30.7606074Z   GPU_FLAG: --gpus all
2022-11-23T03:05:30.7606418Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:05:30.7606745Z ##[endgroup]
2022-11-23T03:05:31.2039288Z NOTE: s3-prefix specified, ignoring name parameter
2022-11-23T03:05:31.2040095Z With the provided path, there will be 1 file uploaded
2022-11-23T03:05:31.2040479Z Uploading to s3 prefix: pytorch/pytorch/3528293562/1/artifact
2022-11-23T03:05:31.2051593Z Starting upload of usage-log-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655200056.zip
2022-11-23T03:05:31.3839708Z Finished upload of usage-log-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9655200056.zip
2022-11-23T03:05:31.4008787Z ##[group]Run # shellcheck disable=SC2156
2022-11-23T03:05:31.4009147Z # shellcheck disable=SC2156
2022-11-23T03:05:31.4009567Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2022-11-23T03:05:31.4022749Z shell: /usr/bin/bash -e {0}
2022-11-23T03:05:31.4023005Z env:
2022-11-23T03:05:31.4023251Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:05:31.4023503Z   GPU_FLAG: --gpus all
2022-11-23T03:05:31.4023858Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:05:31.4024192Z ##[endgroup]
2022-11-23T03:05:31.7097866Z ##[group]Run set -x
2022-11-23T03:05:31.7098279Z set -x
2022-11-23T03:05:31.7098610Z python3 -m pip install -r requirements.txt
2022-11-23T03:05:31.7098961Z python3 -m pip install boto3==1.19.12
2022-11-23T03:05:31.7099342Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-11-23T03:05:31.7112656Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:05:31.7112962Z env:
2022-11-23T03:05:31.7113194Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:05:31.7113461Z   GPU_FLAG: --gpus all
2022-11-23T03:05:31.7113810Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:05:31.7114145Z   AWS_DEFAULT_REGION: us-east-1
2022-11-23T03:05:31.7114407Z   BRANCH: master
2022-11-23T03:05:31.7114658Z   TEST_CONFIG: distributed
2022-11-23T03:05:31.7114893Z   SHARD_NUMBER: 1
2022-11-23T03:05:31.7115207Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T03:05:31.7115555Z   PR_NUMBER: 
2022-11-23T03:05:31.7115795Z   PYTORCH_RETRY_TEST_CASES: 1
2022-11-23T03:05:31.7116087Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-11-23T03:05:31.7116396Z   SHA1: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T03:05:31.7116645Z   TAG: 
2022-11-23T03:05:31.7116874Z   WORKFLOW_ID: 3528293562
2022-11-23T03:05:31.7117294Z   GITHUB_TOKEN: ***
2022-11-23T03:05:31.7117559Z   GHA_WORKFLOW_JOB_ID: 9655200056
2022-11-23T03:05:31.7117800Z ##[endgroup]
2022-11-23T03:05:31.7147150Z + python3 -m pip install -r requirements.txt
2022-11-23T03:05:32.0087622Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T03:05:32.0970790Z Collecting astunparse
2022-11-23T03:05:32.1142038Z   Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
2022-11-23T03:05:32.1500983Z Collecting expecttest
2022-11-23T03:05:32.1541410Z   Downloading expecttest-0.1.4-py3-none-any.whl (6.5 kB)
2022-11-23T03:05:32.1975226Z Collecting future
2022-11-23T03:05:32.2016844Z   Downloading future-0.18.2.tar.gz (829 kB)
2022-11-23T03:05:34.1812714Z Collecting hypothesis
2022-11-23T03:05:34.2046365Z   Downloading hypothesis-6.58.0-py3-none-any.whl (396 kB)
2022-11-23T03:05:35.0397792Z Collecting numpy
2022-11-23T03:05:35.0492982Z   Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
2022-11-23T03:05:35.4799891Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2022-11-23T03:05:35.6035546Z Collecting pyyaml
2022-11-23T03:05:35.6085425Z   Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
2022-11-23T03:05:35.6369224Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2022-11-23T03:05:35.6557256Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2022-11-23T03:05:35.7356032Z Collecting six
2022-11-23T03:05:35.7397273Z   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
2022-11-23T03:05:35.7778088Z Collecting types-dataclasses
2022-11-23T03:05:35.7820266Z   Downloading types_dataclasses-0.6.6-py3-none-any.whl (2.9 kB)
2022-11-23T03:05:35.8284875Z Collecting typing_extensions
2022-11-23T03:05:35.8328292Z   Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)
2022-11-23T03:05:35.8933545Z Collecting sympy
2022-11-23T03:05:35.9005530Z   Downloading sympy-1.10.1-py3-none-any.whl (6.4 MB)
2022-11-23T03:05:36.1319368Z Collecting filelock
2022-11-23T03:05:36.1381992Z   Downloading filelock-3.8.0-py3-none-any.whl (10 kB)
2022-11-23T03:05:36.2370341Z Collecting networkx
2022-11-23T03:05:36.2496776Z   Downloading networkx-2.6.3-py3-none-any.whl (1.9 MB)
2022-11-23T03:05:36.3894604Z Collecting jinja2
2022-11-23T03:05:36.3937670Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2022-11-23T03:05:36.4985131Z Collecting wheel<1.0,>=0.23.0
2022-11-23T03:05:36.5030820Z   Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
2022-11-23T03:05:36.5533139Z Collecting sortedcontainers<3.0.0,>=2.1.0
2022-11-23T03:05:36.5573515Z   Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
2022-11-23T03:05:36.6139911Z Collecting attrs>=19.2.0
2022-11-23T03:05:36.6180206Z   Downloading attrs-22.1.0-py2.py3-none-any.whl (58 kB)
2022-11-23T03:05:36.6953322Z Collecting exceptiongroup>=1.0.0; python_version < "3.11"
2022-11-23T03:05:36.6995255Z   Downloading exceptiongroup-1.0.4-py3-none-any.whl (14 kB)
2022-11-23T03:05:36.7120266Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2022-11-23T03:05:36.7148724Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.12)
2022-11-23T03:05:36.7372161Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24)
2022-11-23T03:05:36.7383463Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2022-11-23T03:05:36.7687485Z Collecting mpmath>=0.19
2022-11-23T03:05:36.7770967Z   Downloading mpmath-1.2.1-py3-none-any.whl (532 kB)
2022-11-23T03:05:36.9341578Z Collecting MarkupSafe>=2.0
2022-11-23T03:05:36.9387673Z   Downloading MarkupSafe-2.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
2022-11-23T03:05:36.9500766Z Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
2022-11-23T03:05:37.1486033Z Installing collected packages: wheel, six, astunparse, expecttest, future, sortedcontainers, attrs, exceptiongroup, hypothesis, numpy, pyyaml, types-dataclasses, typing-extensions, mpmath, sympy, filelock, networkx, MarkupSafe, jinja2
2022-11-23T03:05:37.1783736Z   WARNING: The script wheel is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:05:37.1784575Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:05:37.2292514Z     Running setup.py install for future: started
2022-11-23T03:05:37.9161915Z     Running setup.py install for future: finished with status 'done'
2022-11-23T03:05:38.2391926Z   WARNING: The script hypothesis is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:05:38.2393113Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:05:40.3068210Z   WARNING: The scripts f2py, f2py3 and f2py3.7 are installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:05:40.3068908Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:05:49.4525781Z   WARNING: The script isympy is installed in '/home/ec2-user/.local/bin' which is not on PATH.
2022-11-23T03:05:49.4526504Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-11-23T03:05:50.7519728Z Successfully installed MarkupSafe-2.1.1 astunparse-1.6.3 attrs-22.1.0 exceptiongroup-1.0.4 expecttest-0.1.4 filelock-3.8.0 future-0.18.2 hypothesis-6.58.0 jinja2-3.1.2 mpmath-1.2.1 networkx-2.6.3 numpy-1.21.6 pyyaml-6.0 six-1.16.0 sortedcontainers-2.4.0 sympy-1.10.1 types-dataclasses-0.6.6 typing-extensions-4.4.0 wheel-0.38.4
2022-11-23T03:05:50.8302399Z + python3 -m pip install boto3==1.19.12
2022-11-23T03:05:51.1274277Z Defaulting to user installation because normal site-packages is not writeable
2022-11-23T03:05:52.1079481Z Collecting boto3==1.19.12
2022-11-23T03:05:52.1283192Z   Downloading boto3-1.19.12-py3-none-any.whl (131 kB)
2022-11-23T03:05:52.1866138Z Collecting jmespath<1.0.0,>=0.7.1
2022-11-23T03:05:52.1904789Z   Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
2022-11-23T03:05:52.2439917Z Collecting s3transfer<0.6.0,>=0.5.0
2022-11-23T03:05:52.2478243Z   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
2022-11-23T03:05:53.4537077Z Collecting botocore<1.23.0,>=1.22.12
2022-11-23T03:05:53.4587920Z   Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB)
2022-11-23T03:05:53.7123175Z Collecting python-dateutil<3.0.0,>=2.1
2022-11-23T03:05:53.7163883Z   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
2022-11-23T03:05:53.7360594Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.12)
2022-11-23T03:05:53.7579163Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-11-23T03:05:53.9674404Z Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3
2022-11-23T03:05:54.8709138Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.2
2022-11-23T03:05:54.9255256Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-11-23T03:06:10.1030816Z [scribe] Scribe access token not provided, sending report via boto3...
2022-11-23T03:06:10.1032701Z 
2022-11-23T03:06:10.1033332Z ----- Historic stats comparison result ------
2022-11-23T03:06:10.1033572Z 
2022-11-23T03:06:10.1038094Z     job: linux-bionic-cuda11.6-py3.10-gcc7
2022-11-23T03:06:10.1038454Z     commit: 1cfd3858ac54fe3883534309081631a0a892ba3f
2022-11-23T03:06:10.1038658Z 
2022-11-23T03:06:10.1038867Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-11-23T03:06:10.1039119Z 
2022-11-23T03:06:10.1039224Z     : (master)
2022-11-23T03:06:10.1043765Z     |
2022-11-23T03:06:10.1044481Z     * 1cfd3858ac (HEAD)              total time  3303.02s
2022-11-23T03:06:10.1045258Z     * 26322544b8 (base)  14 reports, total time  4545.22s ± 1821.39s
2022-11-23T03:06:10.1045720Z     * 7f4b4d2827         14 reports, total time  4184.38s ± 2086.66s
2022-11-23T03:06:10.1046510Z     * b50699f247         14 reports, total time  4471.77s ± 1861.64s
2022-11-23T03:06:10.1046921Z     * 8bf8e4d71e         14 reports, total time  4570.74s ± 1842.55s
2022-11-23T03:06:10.1047367Z     * ce856cee7e         14 reports, total time  4577.38s ± 1838.96s
2022-11-23T03:06:10.1047791Z     * 391b593ca2         14 reports, total time  4457.06s ± 1822.27s
2022-11-23T03:06:10.1048219Z     * 5bba783d21         14 reports, total time  4489.09s ± 1854.90s
2022-11-23T03:06:10.1048621Z     * ea920a1115         14 reports, total time  4455.13s ± 1838.12s
2022-11-23T03:06:10.1049040Z     * 74e62a1fef         14 reports, total time  4552.71s ± 1811.91s
2022-11-23T03:06:10.1049461Z     * 00b7d8ef23         14 reports, total time  4180.04s ± 2065.84s
2022-11-23T03:06:10.1049717Z     |
2022-11-23T03:06:10.1049924Z     :
2022-11-23T03:06:10.1050071Z 
2022-11-23T03:06:10.1050241Z Removed  (across  998 suites)     0 tests, totaling     0.00s
2022-11-23T03:06:10.1050593Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-11-23T03:06:10.1050941Z Added    (across   68 suites)   792 tests, totaling +4352.64s
2022-11-23T03:06:10.1634229Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2022-11-23T03:06:10.1634585Z with:
2022-11-23T03:06:10.1634782Z env:
2022-11-23T03:06:10.1635020Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:06:10.1635286Z   GPU_FLAG: --gpus all
2022-11-23T03:06:10.1635612Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:06:10.1635939Z ##[endgroup]
2022-11-23T03:06:10.1662094Z ##[group]Run set -eou pipefail
2022-11-23T03:06:10.1662406Z set -eou pipefail
2022-11-23T03:06:10.1662656Z 
2022-11-23T03:06:10.1662977Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2022-11-23T03:06:10.1663299Z for _ in $(seq 1440); do
2022-11-23T03:06:10.1663595Z     # Break if no ssh session exists anymore
2022-11-23T03:06:10.1663900Z     if [ "$(who)" = "" ]; then
2022-11-23T03:06:10.1664133Z       break
2022-11-23T03:06:10.1664399Z     fi
2022-11-23T03:06:10.1664632Z     echo "."
2022-11-23T03:06:10.1664852Z     sleep 5
2022-11-23T03:06:10.1665086Z done
2022-11-23T03:06:10.1678613Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:06:10.1678899Z env:
2022-11-23T03:06:10.1679143Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:06:10.1679410Z   GPU_FLAG: --gpus all
2022-11-23T03:06:10.1679735Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:06:10.1680063Z ##[endgroup]
2022-11-23T03:06:10.1709384Z Holding runner for 2 hours until all ssh sessions have logged out
2022-11-23T03:06:10.1835815Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-11-23T03:06:10.1836228Z # ignore expansion of "docker ps -q" since it could be empty
2022-11-23T03:06:10.1836579Z # shellcheck disable=SC2046
2022-11-23T03:06:10.1836889Z docker stop $(docker ps -q) || true
2022-11-23T03:06:10.1837207Z # Prune all of the docker images
2022-11-23T03:06:10.1837494Z docker system prune -af
2022-11-23T03:06:10.1849576Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-11-23T03:06:10.1849877Z env:
2022-11-23T03:06:10.1850118Z   GIT_DEFAULT_BRANCH: master
2022-11-23T03:06:10.1850371Z   GPU_FLAG: --gpus all
2022-11-23T03:06:10.1850720Z   DOCKER_CONTAINER_ID: 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:06:10.1851051Z ##[endgroup]
2022-11-23T03:06:11.9405591Z 103428310d4e
2022-11-23T03:06:13.0203694Z Deleted Containers:
2022-11-23T03:06:13.0204071Z 103428310d4e34cc3e15779f302da61e5195488891238b54ec4cf426b9e96e70
2022-11-23T03:06:13.0204307Z 
2022-11-23T03:06:18.3298622Z Deleted Images:
2022-11-23T03:06:18.3299885Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:072aae4a77ed7d3a69ad5683420509c41301b940
2022-11-23T03:06:18.3301674Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7@sha256:3a5626edfb2c43fb24303351be75287af92426b6bb7c6df2defc98f980346c6a
2022-11-23T03:06:18.3302607Z deleted: sha256:e2c63e8434298b5b8922fe396fb22d541e83da3321f8559334df676354c6a90a
2022-11-23T03:06:18.3303354Z deleted: sha256:e97e2654456ae35786d9ff4a73ece4d85ce36ae9bd4e402e5f8c4c41a4b8cb5d
2022-11-23T03:06:18.3304081Z deleted: sha256:0191afefc9967131b7cd6196bee5a1d3a4eba8c24d3e11ff67013ecd0d244f4d
2022-11-23T03:06:18.3304762Z deleted: sha256:cd6998962c740e934e511d315fd0139a2737289173123cd7675b630fe71d0a6f
2022-11-23T03:06:18.3305404Z deleted: sha256:684c9dbbc4faf4388438a99012caaa6e9e9c3ac93f3842ff7b2f4c81c6c66866
2022-11-23T03:06:18.3306101Z deleted: sha256:be75865fc66b386df8a53dd220b7f4fa8464d0c86f06b6fa84e7d5b8fa2b5333
2022-11-23T03:06:18.3307200Z deleted: sha256:9e5281171ccc5aa329fd085f38d4831c13f47e27ea26a9243daf336fc701114a
2022-11-23T03:06:18.3307851Z deleted: sha256:0ba6072392ef0b01b99d45293e62f415e397460b4bf5a00257afb7aa9cfccb14
2022-11-23T03:06:18.3308760Z deleted: sha256:5f0fab79723550908a4149737ce5268ceacba20bc9c1aea35acdb6ff93ba4aa7
2022-11-23T03:06:18.3309431Z deleted: sha256:0c8088138816657b983280a5e4385f5c159c90b6be095bc4972290be20d46c16
2022-11-23T03:06:18.3310080Z deleted: sha256:a9cdd96267ff8adf28efa06db7d37977216a7580ca475239528fff85024f9bcb
2022-11-23T03:06:18.3310719Z deleted: sha256:9abd11e0b20ee19055f20e11ac5a4cc029eee3433686ce8ab9ffb6636269391a
2022-11-23T03:06:18.3311394Z deleted: sha256:cb16cc59b9c802a04fe3283c4a00840d0a3c24128b3620964a7aa927a757d672
2022-11-23T03:06:18.3312057Z deleted: sha256:ed27e40372acea88785f25bcd63f03a56960f00e444e3d5b22e52915e885242b
2022-11-23T03:06:18.3312750Z deleted: sha256:395dfa2cf9efd2fde511c14dbaf706e2efb3ab003af0cd725614b86f10643247
2022-11-23T03:06:18.3313419Z deleted: sha256:ca415181cb076083a9af8e85b901ee24154183e2d4c3960e21aab48260376214
2022-11-23T03:06:18.3314022Z deleted: sha256:b13fc2861b47406c24208813cb5398b911d9bae952f11ed9a411f42e221f8dfc
2022-11-23T03:06:18.3314708Z deleted: sha256:9cbf0b121bab50c1cad2d31b40f6c7c52003ba77877a2ef6d9bc87a2c0b073d2
2022-11-23T03:06:18.3315387Z deleted: sha256:60e157b04ecdbe2bce04795e0fade9ec9aae999065bd410785dcbaedd9778a19
2022-11-23T03:06:18.3316024Z deleted: sha256:5eb96691864f520823a417cd2f3278b4c2ac579490941d6c623865e478828c8b
2022-11-23T03:06:18.3316654Z deleted: sha256:e93d6940ac64ac73f178cc63066fb2c3ab041023d66146b32019cb7860511be5
2022-11-23T03:06:18.3317280Z deleted: sha256:e302e1f04c7e3031f83227f08d6987b02f39a75ac0e741754afad2dc1e265f8a
2022-11-23T03:06:18.3317898Z deleted: sha256:d82cdf793dbcd047c1843326443a1249721e7308a7c6fb3e23fe7331652e7047
2022-11-23T03:06:18.3318570Z deleted: sha256:3edb430c2f9009d4993daf017be01fe272bd3452db11c16e51f7755ac845d410
2022-11-23T03:06:18.3319253Z deleted: sha256:16e8f362c1784e16c1db6b1d0aa4449097e6d646f4c8682a122dea7c4da38aaf
2022-11-23T03:06:18.3319902Z deleted: sha256:7f58576cf19df9f3be9082f2c0ec2fc7010409b97ecb99bae66a10805d752f48
2022-11-23T03:06:18.3320555Z deleted: sha256:88688611a15825ecae20cd8c4032711d2351d2f954a9ebcd4c671b2bdb017df8
2022-11-23T03:06:18.3321226Z deleted: sha256:a46e0b74ccdcd4e2eb07727be3bc1a2c4236b1f88c65e64a50234e8a35932a80
2022-11-23T03:06:18.3321880Z deleted: sha256:b633962159aa14dfe94a149d00f90eecaba6dab960d4011bdf3667a5ee9586db
2022-11-23T03:06:18.3322576Z deleted: sha256:a05c7409499ce8c5d7ffc085772c3910c812ec835dc9145bbbb07b8b3c075235
2022-11-23T03:06:18.3323255Z deleted: sha256:0d63a7de5066f69cd9fd1af8fc47405e880de8f88f5cb16278a1f1ac94d0cd41
2022-11-23T03:06:18.3323914Z deleted: sha256:7d74b4ce1a60334100fccc0917345873714e640160622691b579d64c0ae4640f
2022-11-23T03:06:18.3324530Z deleted: sha256:33aae29ffc4791507bef289cfb1f178909f3fc97a40c618723eeec1f8f5bd80c
2022-11-23T03:06:18.3325223Z deleted: sha256:6ea72b84f0436ed1d288baf124dd38e43bbb89e746ccfd3a4ec420ddced8bbc2
2022-11-23T03:06:18.3325920Z deleted: sha256:04e33e1cfdd5a1b2409b80f5881e6cf7b1810fe975aad4ce7c97b0ff6c0e7b4e
2022-11-23T03:06:18.3326787Z deleted: sha256:df1ef30e86bf04681ecd0728263efe1e98b2eea0a228cef29bd0febfc8bdac2f
2022-11-23T03:06:18.3327468Z deleted: sha256:36a44974e500014175f5e49f50c8afa1ac9c5e8092a8ea99c3c97b7ce9c517d8
2022-11-23T03:06:18.3328127Z deleted: sha256:a31f0224d50d031823b07dbb97f256f6960c87ea3c52ebeceef98febab200451
2022-11-23T03:06:18.3328763Z deleted: sha256:c4beec84548d277aff0487a9a5a8c2b3d577421e3275f36106b778c6edbb9d53
2022-11-23T03:06:18.3329394Z deleted: sha256:bcc7df3b45729f5d1802045954e76e3407d9e07ba6f516de0895d775d00ad7f8
2022-11-23T03:06:18.3330068Z deleted: sha256:84de992a179a16ba619507ec45b04b4c0da3d3fa31cedc8f6beb5aaadd7a232a
2022-11-23T03:06:18.3330745Z deleted: sha256:5011206a0b2edc2a6c68ba41313e7f283ee7c925ab6a731f8818d01352f68596
2022-11-23T03:06:18.3331410Z deleted: sha256:46a56b12ac94daa35c90ac97d26adfde704693e34613d69fb97687aa53ae33f5
2022-11-23T03:06:18.3332067Z deleted: sha256:ed2b7a9e28b3474bc9b7e68f8158ecda88b3fa3d3ab1587898fa976922af0deb
2022-11-23T03:06:18.3332746Z deleted: sha256:4a6976746db7764bb48f2a06af1fb5f88e3646edc1c9bc0d18686d5a6350cac0
2022-11-23T03:06:18.3333558Z deleted: sha256:5e175425e3e9ec93e8c6c1b7560b49ef5e95af68ec55757902072a8dca020323
2022-11-23T03:06:18.3334208Z deleted: sha256:fb740502513c6cf883c844f03760de367c4c70d09a69b9476bcf737b4578563a
2022-11-23T03:06:18.3334849Z deleted: sha256:2c105119fc030d11b3d570ec9a83948a1fb17f138df2a3245f9566b89de51495
2022-11-23T03:06:18.3335492Z deleted: sha256:8caad6b6cba0d0ced7e21fe4b2027b8647d66b7f78c34367dd8571a0520ba2c0
2022-11-23T03:06:18.3336205Z deleted: sha256:1051db32aefad193995ca536ed99e29eed4fd0340ddda721ec11e9c4eb9e93af
2022-11-23T03:06:18.3336867Z deleted: sha256:c6b2a4553f41b3b4a3dc6a26be0020c98980bb4e7186d194901769dce6716c27
2022-11-23T03:06:18.3337550Z deleted: sha256:8faec3528fe75bb31f14d0caf8707a2fe4b70f60d7e631c2b3dbb36cd6d83dd9
2022-11-23T03:06:18.3338234Z deleted: sha256:7574bc80094251ac667e6bed9dd5a808ecf6f61f23c8d4c56a69c644d06f4e32
2022-11-23T03:06:18.3338882Z deleted: sha256:69f57fbceb1b420d7e4697e0f6514887b0805ee0059bea7d51e0a832962e74bf
2022-11-23T03:06:18.3339295Z 
2022-11-23T03:06:18.3459392Z Total reclaimed space: 21.93GB
2022-11-23T03:06:18.3523459Z Post job cleanup.
2022-11-23T03:06:18.3584973Z Post job cleanup.
2022-11-23T03:06:18.4976925Z [command]/usr/bin/git version
2022-11-23T03:06:18.5034142Z git version 2.37.1
2022-11-23T03:06:18.5100588Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/f8f0a03a-2270-48dd-92cf-893aac0fbfbf' before making global git config changes
2022-11-23T03:06:18.5101443Z Adding repository directory to the temporary git global config as a safe directory
2022-11-23T03:06:18.5108416Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-11-23T03:06:18.5153494Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-11-23T03:06:18.5190932Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-11-23T03:06:18.5536062Z Entering 'android/libs/fbjni'
2022-11-23T03:06:18.5578808Z Entering 'third_party/FP16'
2022-11-23T03:06:18.5622587Z Entering 'third_party/FXdiv'
2022-11-23T03:06:18.5663869Z Entering 'third_party/NNPACK'
2022-11-23T03:06:18.5708289Z Entering 'third_party/QNNPACK'
2022-11-23T03:06:18.5752659Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T03:06:18.5807623Z Entering 'third_party/XNNPACK'
2022-11-23T03:06:18.5862817Z Entering 'third_party/benchmark'
2022-11-23T03:06:18.5906362Z Entering 'third_party/cpuinfo'
2022-11-23T03:06:18.5949652Z Entering 'third_party/cub'
2022-11-23T03:06:18.5991393Z Entering 'third_party/cudnn_frontend'
2022-11-23T03:06:18.6059175Z Entering 'third_party/cutlass'
2022-11-23T03:06:18.6109028Z Entering 'third_party/eigen'
2022-11-23T03:06:18.6153134Z Entering 'third_party/fbgemm'
2022-11-23T03:06:18.6194747Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T03:06:18.6237353Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T03:06:18.6281392Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T03:06:18.6324344Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T03:06:18.6370162Z Entering 'third_party/flatbuffers'
2022-11-23T03:06:18.6415312Z Entering 'third_party/fmt'
2022-11-23T03:06:18.6456967Z Entering 'third_party/foxi'
2022-11-23T03:06:18.6499163Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T03:06:18.6541629Z Entering 'third_party/gloo'
2022-11-23T03:06:18.6585897Z Entering 'third_party/googletest'
2022-11-23T03:06:18.6628921Z Entering 'third_party/ideep'
2022-11-23T03:06:18.6670554Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T03:06:18.6715145Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T03:06:18.6764598Z Entering 'third_party/ios-cmake'
2022-11-23T03:06:18.6806117Z Entering 'third_party/ittapi'
2022-11-23T03:06:18.6848807Z Entering 'third_party/kineto'
2022-11-23T03:06:18.6890337Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T03:06:18.6932564Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T03:06:18.6975143Z Entering 'third_party/nccl/nccl'
2022-11-23T03:06:18.7016899Z Entering 'third_party/neon2sse'
2022-11-23T03:06:18.7058220Z Entering 'third_party/nlohmann'
2022-11-23T03:06:18.7101637Z Entering 'third_party/onnx'
2022-11-23T03:06:18.7157686Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T03:06:18.7200802Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T03:06:18.7245942Z Entering 'third_party/onnx-tensorrt'
2022-11-23T03:06:18.7287358Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T03:06:18.7334613Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T03:06:18.7377149Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T03:06:18.7418757Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T03:06:18.7465890Z Entering 'third_party/pocketfft'
2022-11-23T03:06:18.7507854Z Entering 'third_party/protobuf'
2022-11-23T03:06:18.7557042Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T03:06:18.7600358Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T03:06:18.7646459Z Entering 'third_party/psimd'
2022-11-23T03:06:18.7690201Z Entering 'third_party/pthreadpool'
2022-11-23T03:06:18.7731821Z Entering 'third_party/pybind11'
2022-11-23T03:06:18.7773695Z Entering 'third_party/python-enum'
2022-11-23T03:06:18.7817683Z Entering 'third_party/python-peachpy'
2022-11-23T03:06:18.7860578Z Entering 'third_party/python-six'
2022-11-23T03:06:18.7902277Z Entering 'third_party/sleef'
2022-11-23T03:06:18.7944366Z Entering 'third_party/tbb'
2022-11-23T03:06:18.7988968Z Entering 'third_party/tensorpipe'
2022-11-23T03:06:18.8032352Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T03:06:18.8074605Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T03:06:18.8116465Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T03:06:18.8158667Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T03:06:18.8199860Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T03:06:18.8244110Z Entering 'third_party/zstd'
2022-11-23T03:06:18.8303665Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-11-23T03:06:18.8334048Z http.https://github.com/.extraheader
2022-11-23T03:06:18.8343937Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-11-23T03:06:18.8380928Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-11-23T03:06:18.8699578Z Entering 'android/libs/fbjni'
2022-11-23T03:06:18.8725360Z http.https://github.com/.extraheader
2022-11-23T03:06:18.8758433Z Entering 'third_party/FP16'
2022-11-23T03:06:18.8783234Z http.https://github.com/.extraheader
2022-11-23T03:06:18.8816297Z Entering 'third_party/FXdiv'
2022-11-23T03:06:18.8841638Z http.https://github.com/.extraheader
2022-11-23T03:06:18.8873962Z Entering 'third_party/NNPACK'
2022-11-23T03:06:18.8898515Z http.https://github.com/.extraheader
2022-11-23T03:06:18.8931099Z Entering 'third_party/QNNPACK'
2022-11-23T03:06:18.8956109Z http.https://github.com/.extraheader
2022-11-23T03:06:18.8988686Z Entering 'third_party/VulkanMemoryAllocator'
2022-11-23T03:06:18.9013529Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9046442Z Entering 'third_party/XNNPACK'
2022-11-23T03:06:18.9071369Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9115032Z Entering 'third_party/benchmark'
2022-11-23T03:06:18.9139418Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9172107Z Entering 'third_party/cpuinfo'
2022-11-23T03:06:18.9197912Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9230901Z Entering 'third_party/cub'
2022-11-23T03:06:18.9255528Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9287971Z Entering 'third_party/cudnn_frontend'
2022-11-23T03:06:18.9313205Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9351691Z Entering 'third_party/cutlass'
2022-11-23T03:06:18.9376203Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9418682Z Entering 'third_party/eigen'
2022-11-23T03:06:18.9443558Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9478330Z Entering 'third_party/fbgemm'
2022-11-23T03:06:18.9503924Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9536321Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-11-23T03:06:18.9560609Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9592675Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-11-23T03:06:18.9616444Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9648871Z Entering 'third_party/fbgemm/third_party/googletest'
2022-11-23T03:06:18.9673833Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9706162Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-11-23T03:06:18.9730679Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9764343Z Entering 'third_party/flatbuffers'
2022-11-23T03:06:18.9790360Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9826186Z Entering 'third_party/fmt'
2022-11-23T03:06:18.9850821Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9882957Z Entering 'third_party/foxi'
2022-11-23T03:06:18.9907882Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9940709Z Entering 'third_party/gemmlowp/gemmlowp'
2022-11-23T03:06:18.9966434Z http.https://github.com/.extraheader
2022-11-23T03:06:18.9998967Z Entering 'third_party/gloo'
2022-11-23T03:06:19.0023346Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0055673Z Entering 'third_party/googletest'
2022-11-23T03:06:19.0081365Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0114239Z Entering 'third_party/ideep'
2022-11-23T03:06:19.0138758Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0172401Z Entering 'third_party/ideep/mkl-dnn'
2022-11-23T03:06:19.0197846Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0233543Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-11-23T03:06:19.0257512Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0296720Z Entering 'third_party/ios-cmake'
2022-11-23T03:06:19.0323508Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0356946Z Entering 'third_party/ittapi'
2022-11-23T03:06:19.0383496Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0416020Z Entering 'third_party/kineto'
2022-11-23T03:06:19.0441963Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0474770Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-11-23T03:06:19.0499211Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0532617Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-11-23T03:06:19.0556971Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0592040Z Entering 'third_party/nccl/nccl'
2022-11-23T03:06:19.0617396Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0650606Z Entering 'third_party/neon2sse'
2022-11-23T03:06:19.0676834Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0709139Z Entering 'third_party/nlohmann'
2022-11-23T03:06:19.0733719Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0767931Z Entering 'third_party/onnx'
2022-11-23T03:06:19.0793252Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0841468Z Entering 'third_party/onnx/third_party/benchmark'
2022-11-23T03:06:19.0866424Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0899863Z Entering 'third_party/onnx/third_party/pybind11'
2022-11-23T03:06:19.0924313Z http.https://github.com/.extraheader
2022-11-23T03:06:19.0960490Z Entering 'third_party/onnx-tensorrt'
2022-11-23T03:06:19.0984824Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1017241Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-11-23T03:06:19.1042186Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1080550Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-11-23T03:06:19.1105164Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1137573Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-11-23T03:06:19.1162982Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1195603Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-11-23T03:06:19.1219882Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1257789Z Entering 'third_party/pocketfft'
2022-11-23T03:06:19.1283048Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1315404Z Entering 'third_party/protobuf'
2022-11-23T03:06:19.1340696Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1377734Z Entering 'third_party/protobuf/third_party/benchmark'
2022-11-23T03:06:19.1402459Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1435184Z Entering 'third_party/protobuf/third_party/googletest'
2022-11-23T03:06:19.1459140Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1494245Z Entering 'third_party/psimd'
2022-11-23T03:06:19.1519392Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1552106Z Entering 'third_party/pthreadpool'
2022-11-23T03:06:19.1576553Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1609648Z Entering 'third_party/pybind11'
2022-11-23T03:06:19.1634608Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1667125Z Entering 'third_party/python-enum'
2022-11-23T03:06:19.1691812Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1723942Z Entering 'third_party/python-peachpy'
2022-11-23T03:06:19.1748811Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1781044Z Entering 'third_party/python-six'
2022-11-23T03:06:19.1805989Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1838363Z Entering 'third_party/sleef'
2022-11-23T03:06:19.1863107Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1895327Z Entering 'third_party/tbb'
2022-11-23T03:06:19.1920176Z http.https://github.com/.extraheader
2022-11-23T03:06:19.1954749Z Entering 'third_party/tensorpipe'
2022-11-23T03:06:19.1980400Z http.https://github.com/.extraheader
2022-11-23T03:06:19.2013490Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-11-23T03:06:19.2038942Z http.https://github.com/.extraheader
2022-11-23T03:06:19.2071436Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-11-23T03:06:19.2095990Z http.https://github.com/.extraheader
2022-11-23T03:06:19.2128045Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-11-23T03:06:19.2152086Z http.https://github.com/.extraheader
2022-11-23T03:06:19.2184121Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-11-23T03:06:19.2209587Z http.https://github.com/.extraheader
2022-11-23T03:06:19.2241246Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-11-23T03:06:19.2265218Z http.https://github.com/.extraheader
2022-11-23T03:06:19.2300407Z Entering 'third_party/zstd'
2022-11-23T03:06:19.2325605Z http.https://github.com/.extraheader
2022-11-23T03:06:19.2621617Z Cleaning up orphan processes